henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
1,589 Commits
37 Branches
0 Tags
20 MiB
mlang/suspicious-skew
master
mike/encode-worker
mike/playlist-debug
mike/faster-chat-merge
HubbeKing-multiplatform-builds
thrimbletrimmer-solid
bus-progress-ui
bus-post-processing
chrusher/bus_synthesizer
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '41f9dd9a24'
${ noResults }
Commit Graph

1589 Commits (41f9dd9a2410f8f64e66b3931479d6b23c0994c6)
 

Author SHA1 Message Date
Mike Lang 41f9dd9a24
Merge 50c233b58c into d7facca842 2 weeks ago
Mike Lang d7facca842 pubbot: Change donation firehose stream 3 weeks ago
Mike Lang a6a2ca1a96 fix api_ping as gevent now somehow fails with this pinned version 
```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.11/site-packages/api_ping/__main__.py", line 3, in <module>
    gevent.monkey.patch_all()
  File "/usr/lib/python3.11/site-packages/gevent/monkey.py", line 1255, in patch_all
    _notify_patch(events.GeventWillPatchAllEvent(modules_to_patch, kwargs), _warnings)
  File "/usr/lib/python3.11/site-packages/gevent/monkey.py", line 190, in _notify_patch
    notify_and_call_entry_points(event)
  File "/usr/lib/python3.11/site-packages/gevent/events.py", line 104, in notify_and_call_entry_points
    subscriber = plugin.load()
                 ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2745, in load
    self.require(*args, **kwargs)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2773, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 889, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 930, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'zope.event' distribution was not found and is required by the application
```

god i hate python packaging.
 2 months ago
Mike Lang 58f85b30ac youtubebot: Show the video title 
This is an additional 1 quota per video, but with a cache (it's ok if it's out of date)
this should be no issue.
 2 months ago
Mike Lang 094edfc701 downloader: support 2k/4k twitch streams 
You need to explicitly say you support x265
 2 months ago
Mike Lang 090a0af7e6 sheetsync: don't report errors on non-image media 3 months ago
Mike Lang b1dbb4f042 give up on timezone crap and use a hard-coded number again 
the python version in our container has no timezone data?
 3 months ago
Mike Lang 2232bafe95 fix stupid timezone crap 3 months ago
Mike Lang a2403b56cd schedulebot: fix time zones 3 months ago
Mike Lang c53c5a98d5 pubbot: work in years besides 2024 3 months ago
Mike Lang 2c423f65fb prizebot: Fix year regex 3 months ago
Mike Lang 6b7ae95c1f prizebot: make it work for years besides 2024 3 months ago
Mike Lang 15f4767f32 fix another issue with the thing i yolo'd into prod 3 months ago
Mike Lang b24612d271 fix typo in prev commit 3 months ago
Mike Lang 67d351e549 cutter: Add mode where it only uploads rows with uploader_whitelist set 
So you can have a "special" cutter that doesn't upload most videos
 3 months ago
Mike Lang 54fd356b39 Add youtubebot 
This adds a zulip bot that polls the youtube API for new comment threads, and posts them to Zulip.
Some limitations:
- It doesn't keep any state, so it won't post anything it "missed" while not running.
- It can only find top-level comments, not replies
- For quota reasons, we shouldn't poll more often than every 1 minute (at this rate we consume approx 1 upload worth of quota per day)
- If somehow there are more than 100 comments within 1 minute, it will miss all but the last 100.
 3 months ago
Dan Collins 3d9490d335 If an empty title/desc was saved, keep it rather than using the default 4 months ago
Dan Collins a04a318ea8 Thrimbletrimmer: Default title to blank to encourage editors to write their own 
This patch simply uses the RDP tag to decide whether to use the row
description as a default for the video title, or for the video
description. This will probably be changed by #479 down the line,
but that's lower priority and more complex, so this is a good
starting point.

Resolves #504
 4 months ago
Dan Collins e04c76e0f8
Thrimbletrimmer: Add keyboard shortcuts for +/- 1 minute (#520) 
This patch adds Shift+Arrow Key keyboard shortcuts to move forward or backwards by 1 minute in Thrimbletrimmer, allowing for faster navigation.

Resolves #509
 4 months ago
Dan Collins 81a9cd7e7c #486: Prevent arrow keys on the crop images from moving the play head 4 months ago
Mike Lang 50c233b58c fix copy paste error in v3/v4 and benchmark duplicate merges 
Merging 1000 duplicates of the same batch:
v1: 98ms
v2: 101ms
v3: 116ms
v4: 116ms

So as expected v3 and v4 have a small cost with duplicates for optimizing for non-overlapping,
but even with ridiculous numbers duplicates are fast to process anyway.
 5 months ago
Mike Lang c87a25e875 merge_v4 is more specialized to merge large numbers of non-overlapping batches 
It can do 24h of batches in 180ms, ~180x faster than v1.

I'm a little concerned it's overly specialized for that role and will be slow to merge duplicate batches,
but those operations involve tiny amounts of data by comparison so it doesn't really matter.
 5 months ago
Mike Lang eeeb48c0f8 add third implementation 
which is really just the second implementation but generalized to N-way merge instead of 2-way.

Basic benchmark results:
                         v1   v2   v3
    1 hour (60 batches): 19ms 31ms 23ms
24 hours (1440 batches): 34s  130s 17s

So it's about as fast as v1 when it doesn't matter, and twice as fast when it does.
 5 months ago
Mike Lang 91e3e54a14 add ability to dump output to debug 5 months ago
Mike Lang 9e6e20cee2 schema: Create a type for image areas 
This creates a new postgres "domain" which is a subtype of some other type
(in this case, of INTEGER[]) with some constraints applied.
We use this to create a type which is explicitly an array of length 4.

This saves us from needing to repeat this constraint everywhere (which already led to one copy-paste error)
and is arguably clearer in intent.
 6 months ago
Mike Lang de3e85c893 Fixes to make postgres 17 work 
- WUBLOADER_USER now explicitly requires a grant on the public schema to create tables
- `wal_keep_segments` renamed to `wal_keep_size` and is now a file size, not a number of segments
- Remove some other config entries which did nothing or matched defaults
 6 months ago
Christopher Usher fed653dfbd Updated Postgres version 6 months ago
Mike Lang 0a638b0534 postgres: fix wrong constraint on thumbnail_location values 6 months ago
Mike Lang bbee33afb2 It's ben 9 months ago
Mike Lang aebbb603fc bus_analyzer: use 0.1s before the end timestamp, not the exact end 9 months ago
Mike Lang dc291d4e64 bus_analyzer: check for new segments more often 
to lower latency
 9 months ago
Mike Lang 5db7bcda71 bus_analyzer: use last frame of segment, not first 9 months ago
Mike Lang 941050aea9 end of run memes 9 months ago
Mike Lang 6062f0a8ec restreamer: Ignore transitions when doing rough cuts, instead of disallowing 9 months ago
Christopher Usher 8a0ca215cf Updated colour of the dawn sky and dashboard to observed value 9 months ago
Mike Lang 2aadf79bfb thrimshim odo hack: assume unmatched time of day means dawn 
until we can fix dawn detection.
 9 months ago
Mike Lang bf9da27ca4 restreamer: refuse to load more than 2h of chat 
this hard locks up the server due to merge taking a very long time
 9 months ago
Mike Lang cb08f49003 pubbot: update total var before sending to zulip 
so if zulip is down it still saves
 9 months ago
Mike Lang 5012c2cee2 prizebot: fix urls 9 months ago
ElementalAlchemist c38144d478 Build bus_analyzer 9 months ago
Mike Lang 8ccb6a6e60 pubbot: Post prize names 9 months ago
Mike Lang 61d6df5d1f drive clock: Avoid using "sky" tag ids as uBO blocks them 9 months ago
Mike Lang c54ed3455c driveclock: Run initial update immediately 9 months ago
Mike Lang 91bce47be1 Add driveclock to thrimbletrimmer 9 months ago
Mike Lang e794b0cf7d prizebot fixes 9 months ago
Mike Lang 968e6ec273 make prizebot work with new website 9 months ago
Mike Lang 3b1c837bfa Import prizebot 9 months ago
Mike Lang 9e8cb7bee8 Make ensure_directory() work with paths with no directory component 9 months ago
Mike Lang 854b5723bb Fix wrong variable names in archive cutting 9 months ago
Mike Lang b88627183d Fix modifying manually-uploaded links 9 months ago