henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
1,158 Commits
37 Branches
0 Tags
20 MiB
mlang/suspicious-skew
master
mike/encode-worker
mike/playlist-debug
mike/faster-chat-merge
HubbeKing-multiplatform-builds
thrimbletrimmer-solid
bus-progress-ui
bus-post-processing
chrusher/bus_synthesizer
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'f6b69fef16'
${ noResults }
Commit Graph

1158 Commits (f6b69fef160907b3ecec521d1e2978bd041b67a0)
 

Author SHA1 Message Date
ZeldaZach f6b69fef16 PR #396: Marker commit for authorship info 
This entire PR (of which this commit is the capstone) was co-authored by @ZeldaZach.
In the process of editing his code the original authorship information was lost,
and it ended up being a mix of our code that made it into the final PR.

As a way to ensure he is still at least somewhat credited for this commit series,
this empty commit lists him as author so he is counted as a contributor in Github stats.
 1 year ago
Mike Lang a6dee895f3 playlist_manager: Improved handling of playlist outdated errors 
By restarting the update immediately instead of waiting for the next one.
We only try this up to 3 times to prevent excessive quota usage if it keeps happening.
 1 year ago
Mike Lang 83bf2a4adc playlist_manager: Check for re-orderings without doing them initially 
Previously, we would only do reorderings if we were refreshing the playlist
for some other reason (eg. a video was inserted).

We want to refresh the playlist before attempting reorderings,
so we split the routine into two parts:
- A part that finds out-of-order videos and returns a list of moves to make.
- A part that executes those moves.

We do the former before AND after refreshing, and the latter only with the result from after.
 1 year ago
Mike Lang 0f7fb7f299 playlist_manager: Reorder first/last videos when needed 1 year ago
Mike Lang ca3c506fb4 playlist_manager: Make a specialized error for api requests 
So we can easily check the status code.
 1 year ago
Mike Lang 9150e895fd playlist_manager: Sort first/last videos correctly 
In addition to sorting the videos themselves into the correct spots,
we need to special case them when scanning for the correct place to insert other videos.

Note this only places them in the correct place on insert,
which requires they be set in the playlist config BEFORE being inserted.

A follow-up commit will handle the case of needing to re-order them post insert.
 1 year ago
Mike Lang 93422aa811 playlist_manager: Fetch first/last event info when fetching playlists 
And switch to passing around a namedtuple of these + tags instead of just tags.

To avoid confusion with the list of videos in the playlist, we refer to this data as the "playlist config".
 1 year ago
Mike Lang 580c4c7a90 schema: Add first and last event ids to playlist table 
These represent a pinned first/last video in a playlist.
On the choice of a video id vs an event id:
- Event ids are known before video ids, so we can "set and forget" before a video is uploaded
- No need to re-set if an event's video is re-edited or changed
- In cases where an external video is desired, we can use manual link to associate an event with it

Since we're referencing a primary key, we might as well also make it a proper foreign key
with sensible delete behaviour, though in practice we never delete events.
 1 year ago
Mike Lang 4426a68230 playlist_manager: Cache playlist entry IDs along with video IDs 
This is required in order to be able to move entries later.
Note our view of entry IDs may always be out of date, so any time you use one
you have to handle it no longer existing.
 1 year ago
Mike Lang 84672aa156 playlist_manager: Wrap lookups of playlist_state into a function 
This prevents accidents where we do a simple lookup instead of a .get()
 1 year ago
Mike Lang 9f3989aa3a playlist_manager: Convert format() to f-strings 
Just some general cleanup.
 1 year ago
Mike Lang b952397efc playlist_manager: Explicitly say when a variable is a "playlist_id" vs a "playlist" (a list of videos) 
This makes the code much easier to read by making it clear what each variable actually refers to.
 1 year ago
Mike Lang b119d4bb02 playlist_manager: Clarify set operations 
Use slightly more verbose opertations with intermediate variables that make it clear what we're doing
 1 year ago
Mike Lang 457cec0f57 playlist_manager: Fix typo in comment 1 year ago
Mike Lang a34af372d0
thrimbletrimmer: Validate chapter titles are ascii only 
We are not sure what characters are allowed in chapter titles.
Emoji seem to be disallowed. It is unknown whether things like accents or smart quotes are allowed.
To be conservative, we warn if there are any non-ascii characters in the chapter title.
 1 year ago
Mike Lang cb144866a6 Change wording of "submit anyway" button when saving a draft 1 year ago
Mike Lang f9ff537b84 restreamer: Fix a bug where playlist reports video is finished spuriously 
This happens when we are live viewing a stream, and the last available segment
is at the end of an hour.
We generate the end timestamp as being the end of the last available hour,
which might be within the range of the last available segment. When this happens
we stream the last segment then say we reached the requested end point.
This makes the player stop asking for more segments.

The fix is to pad the end time by an extra hour so we're asking for 1 hour more than the
last available hour.
 1 year ago
Mike Lang daf6001402 restreamer: Move get_best_segments() inside playlist cache 1 year ago
Mike Lang 264545eb9d CachedIterator: Fix bug where state can change while taking the lock 
Resulting in a case where we grab the wrong result, or even try to get the next item
after the iterator has already been discarded.
 1 year ago
Mike Lang 0895ce44ff generate media playlist: Yield in chunks for performance 
Flask sends a chunked response with one chunk per item yielded.
This adds a lot of overhead per yielded item.
We avoid this by collecting the lines of the media playlist into larger chunks
and only flushing once every 1000 segments.

For small playlists this means they'll be emitted as one chunk,
but for large playlists we still get the streaming behaviour.
 1 year ago
Mike Lang 1857a998c9 reduce overhead of gevent.idle() by only yielding once per 1000 segments 1 year ago
Mike Lang 7e90346b87 restreamer: Have playlist cache stream results 1 year ago
Mike Lang 8ede4622ca CachedIterator: Re-serve any errors encountered while iterating 
instead of the second one to reach the error treating it as a successful end of iterator.
 1 year ago
Mike Lang 909cece196 restreamer: Move media playlist cache to also cover HLS playlist generation 1 year ago
Mike Lang cb2c58c75c restreamer: Yield the event loop during playlist generation 1 year ago
Mike Lang be7a9cd1fa restreamer: Make media playlist streamed 
To deal better with very large playlists
 1 year ago
Mike Lang a6d4f2466a restreamer: coalesce concurrent requests for the same playlist URL 1 year ago
Christopher Usher 28a468477d
Update DATABASE.md 
Fixed some minor formatting issues
 1 year ago
HubbeKing b46d9f30bb Proper example for edit_url 2 years ago
HubbeKing 7715a9c702 Run chat_archiver by default and backfill emotes by default 2 years ago
HubbeKing c9af44c327 Always create credentials secret 2 years ago
HubbeKing a98fea2489 Update NFS mount options 2 years ago
HubbeKing 517b881674 Remove old URL from edit_url config setting 2 years ago
HubbeKing 209cc6d876 Add config parameter for NFS mount options 2 years ago
HubbeKing 17596655f6 Update k8s.jsonnet file to support 2023 changes to images 2 years ago
Mike Lang 2a1f7207a8 Allow a fudge factor when checking for gaps/overlaps between segments 
Sometimes in the wild (particularly on youtube) segments may not be timed perfectly, so allow up to 10ms of gap or overlap
to be counted as "equal" for purposes of finding the best segment.
 2 years ago
Mike Lang b8cf0f20b0 segment_coverage: Allow alternate output location + single-run mode 2 years ago
Mike Lang 3606fadaa8 Pin gevent version to work around build issues 
Seeing the following error on latest versions of gevent:

 Traceback (most recent call last):
   File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
     return _run_code(code, main_globals, None,
   File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
     exec(code, run_globals)
   File "/usr/lib/python3.9/site-packages/zulip_bots/schedulebot.py", line 2, in <module>
     import gevent.monkey
   File "/usr/lib/python3.9/site-packages/gevent/__init__.py", line 72, in <module>
     from gevent._hub_local import get_hub
   File "/usr/lib/python3.9/site-packages/gevent/_hub_local.py", line 150, in <module>
     import_c_accel(globals(), 'gevent.__hub_local')
   File "/usr/lib/python3.9/site-packages/gevent/_util.py", line 148, in import_c_accel
     mod = importlib.import_module(cname)
   File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'gevent._gevent_c_hub_local'
 2 years ago
Mike Lang 859852fc50 memes 2 years ago
Christopher Usher b7e8b83c8d corrected shift names 2 years ago
Christopher Usher e12e361525 Shifts graphs 2 years ago
Christopher Usher e66540087a Fixes suggested by ekim 2 years ago
Christopher Usher eae430d5f3 removed local script 2 years ago
Christopher Usher 9a3b1fd5c3 Refactor in preperation for multiple graphs 2 years ago
Christopher Usher a6e990db68 Donation graph now works in a container 2 years ago
Christopher Usher 2c8a27330f graphs container now builds 2 years ago
Christopher Usher 3e8b4dfcf1 Starting integration of donation graphs 2 years ago
Mike Lang c65eb2eae3 Add a default timeout on google APIs 2 years ago
Mike Lang b9b7a6ca27 more schedulebot formatting 2 years ago
Mike Lang 4786154bca One more schedulebot format tweak 2 years ago