henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
1,589 Commits
37 Branches
0 Tags
20 MiB
mlang/suspicious-skew
master
mike/encode-worker
mike/playlist-debug
mike/faster-chat-merge
HubbeKing-multiplatform-builds
thrimbletrimmer-solid
bus-progress-ui
bus-post-processing
chrusher/bus_synthesizer
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '871a2e1909'
${ noResults }
Commit Graph

1589 Commits (871a2e190942212c5b1dd3a44ab8c2754df5ce42)
 

Author SHA1 Message Date
Mike Lang 871a2e1909 Fix connection pool warnings by increasing pool size 
in backfiller and downloader, the things making lots of outgoing http requests.

We want these larger sizes anyway to improve performance in downloader and backfiller.
 4 days ago
Mike Lang 8eec87355c backfiller: Don't backfill segments that only differ from existing segment by 1ms 
In the wild we've seen different servers get timestamps that differ by 1ms for segments
that are otherwise identical - same content, same duration.

The allowable fudge factor for segments is already 10ms, so having timing be 1ms different
between servers shouldn't cause any problems.
Worst case, there's a slight chance you'll get an adjacent frame when picking a cut point / thumbnail.
 4 days ago
Mike Lang c13aa26726 Don't have restreamer respond with metrics to non-restreamer metrics requests 6 days ago
Mike Lang a0c0698446 playlist debug fix 1 week ago
Mike Lang ebc5afd58b Temporary debugging: Write out last 10 media playlists after fetching each segment 1 week ago
Mike Lang 37e225adab more debugging logs 1 week ago
Mike Lang 2ecd4e0a3e more metrics for tracking skew 1 week ago
Mike Lang 663449498c downloader: Re-connect when we see a time error over 0.01s 
We have observed an issue on twitch where there will be a small time jump
(eg. with 2s segments, times will be 00:10.3, 00:12.3, 00:14.7, 00:16.7)
and all subsequent segment timestamps will be out by this amount compared
to what other downloader instances see. Our workaround for this issue is to
watch for such gaps and:
1. trigger a worker refresh (which seems to fix the issue)
2. treat all subsequent segments as "suspect" so they are still saved
but only used if no other source is available.
 1 week ago
Mike Lang d7facca842 pubbot: Change donation firehose stream 1 week ago
Mike Lang a6a2ca1a96 fix api_ping as gevent now somehow fails with this pinned version 
```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.11/site-packages/api_ping/__main__.py", line 3, in <module>
    gevent.monkey.patch_all()
  File "/usr/lib/python3.11/site-packages/gevent/monkey.py", line 1255, in patch_all
    _notify_patch(events.GeventWillPatchAllEvent(modules_to_patch, kwargs), _warnings)
  File "/usr/lib/python3.11/site-packages/gevent/monkey.py", line 190, in _notify_patch
    notify_and_call_entry_points(event)
  File "/usr/lib/python3.11/site-packages/gevent/events.py", line 104, in notify_and_call_entry_points
    subscriber = plugin.load()
                 ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2745, in load
    self.require(*args, **kwargs)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2773, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 889, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 930, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'zope.event' distribution was not found and is required by the application
```

god i hate python packaging.
 1 month ago
Mike Lang 58f85b30ac youtubebot: Show the video title 
This is an additional 1 quota per video, but with a cache (it's ok if it's out of date)
this should be no issue.
 1 month ago
Mike Lang 094edfc701 downloader: support 2k/4k twitch streams 
You need to explicitly say you support x265
 2 months ago
Mike Lang 090a0af7e6 sheetsync: don't report errors on non-image media 2 months ago
Mike Lang b1dbb4f042 give up on timezone crap and use a hard-coded number again 
the python version in our container has no timezone data?
 2 months ago
Mike Lang 2232bafe95 fix stupid timezone crap 2 months ago
Mike Lang a2403b56cd schedulebot: fix time zones 2 months ago
Mike Lang c53c5a98d5 pubbot: work in years besides 2024 2 months ago
Mike Lang 2c423f65fb prizebot: Fix year regex 2 months ago
Mike Lang 6b7ae95c1f prizebot: make it work for years besides 2024 2 months ago
Mike Lang 15f4767f32 fix another issue with the thing i yolo'd into prod 2 months ago
Mike Lang b24612d271 fix typo in prev commit 2 months ago
Mike Lang 67d351e549 cutter: Add mode where it only uploads rows with uploader_whitelist set 
So you can have a "special" cutter that doesn't upload most videos
 2 months ago
Mike Lang 54fd356b39 Add youtubebot 
This adds a zulip bot that polls the youtube API for new comment threads, and posts them to Zulip.
Some limitations:
- It doesn't keep any state, so it won't post anything it "missed" while not running.
- It can only find top-level comments, not replies
- For quota reasons, we shouldn't poll more often than every 1 minute (at this rate we consume approx 1 upload worth of quota per day)
- If somehow there are more than 100 comments within 1 minute, it will miss all but the last 100.
 2 months ago
Dan Collins 3d9490d335 If an empty title/desc was saved, keep it rather than using the default 4 months ago
Dan Collins a04a318ea8 Thrimbletrimmer: Default title to blank to encourage editors to write their own 
This patch simply uses the RDP tag to decide whether to use the row
description as a default for the video title, or for the video
description. This will probably be changed by #479 down the line,
but that's lower priority and more complex, so this is a good
starting point.

Resolves #504
 4 months ago
Dan Collins e04c76e0f8
Thrimbletrimmer: Add keyboard shortcuts for +/- 1 minute (#520) 
This patch adds Shift+Arrow Key keyboard shortcuts to move forward or backwards by 1 minute in Thrimbletrimmer, allowing for faster navigation.

Resolves #509
 4 months ago
Dan Collins 81a9cd7e7c #486: Prevent arrow keys on the crop images from moving the play head 4 months ago
Mike Lang 9e6e20cee2 schema: Create a type for image areas 
This creates a new postgres "domain" which is a subtype of some other type
(in this case, of INTEGER[]) with some constraints applied.
We use this to create a type which is explicitly an array of length 4.

This saves us from needing to repeat this constraint everywhere (which already led to one copy-paste error)
and is arguably clearer in intent.
 5 months ago
Mike Lang de3e85c893 Fixes to make postgres 17 work 
- WUBLOADER_USER now explicitly requires a grant on the public schema to create tables
- `wal_keep_segments` renamed to `wal_keep_size` and is now a file size, not a number of segments
- Remove some other config entries which did nothing or matched defaults
 5 months ago
Christopher Usher fed653dfbd Updated Postgres version 5 months ago
Mike Lang 0a638b0534 postgres: fix wrong constraint on thumbnail_location values 6 months ago
Mike Lang bbee33afb2 It's ben 9 months ago
Mike Lang aebbb603fc bus_analyzer: use 0.1s before the end timestamp, not the exact end 9 months ago
Mike Lang dc291d4e64 bus_analyzer: check for new segments more often 
to lower latency
 9 months ago
Mike Lang 5db7bcda71 bus_analyzer: use last frame of segment, not first 9 months ago
Mike Lang 941050aea9 end of run memes 9 months ago
Mike Lang 6062f0a8ec restreamer: Ignore transitions when doing rough cuts, instead of disallowing 9 months ago
Christopher Usher 8a0ca215cf Updated colour of the dawn sky and dashboard to observed value 9 months ago
Mike Lang 2aadf79bfb thrimshim odo hack: assume unmatched time of day means dawn 
until we can fix dawn detection.
 9 months ago
Mike Lang bf9da27ca4 restreamer: refuse to load more than 2h of chat 
this hard locks up the server due to merge taking a very long time
 9 months ago
Mike Lang cb08f49003 pubbot: update total var before sending to zulip 
so if zulip is down it still saves
 9 months ago
Mike Lang 5012c2cee2 prizebot: fix urls 9 months ago
ElementalAlchemist c38144d478 Build bus_analyzer 9 months ago
Mike Lang 8ccb6a6e60 pubbot: Post prize names 9 months ago
Mike Lang 61d6df5d1f drive clock: Avoid using "sky" tag ids as uBO blocks them 9 months ago
Mike Lang c54ed3455c driveclock: Run initial update immediately 9 months ago
Mike Lang 91bce47be1 Add driveclock to thrimbletrimmer 9 months ago
Mike Lang e794b0cf7d prizebot fixes 9 months ago
Mike Lang 968e6ec273 make prizebot work with new website 9 months ago
Mike Lang 3b1c837bfa Import prizebot 9 months ago