henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
345 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'd6ed138db8'
${ noResults }
Commit Graph

24 Commits (d6ed138db8d9b4c67d0ec8d7c582090b2ef71bcd)

Author SHA1 Message Date
Mike Lang d6ed138db8 cutter: Set an error message to warn humans when it discards a candidate due to holes 
This prevents videos being stuck in EDITED with no visible problem when
they contain holes, but is likely to false positive sometimes.

This is fine though, as it's just a human-readable warning and
it will be cleared as soon as any node accepts the row to be cut.
 5 years ago
Christopher Usher 9ed8facce8 fixed bug in error handling 5 years ago
Christopher Usher a63e18c0a9 added state to upload_errors 5 years ago
Christopher Usher 228468749c changes based on ekim's suggestions 5 years ago
Christopher Usher 5c08540c0a stats for the cutter 5 years ago
Christopher Usher 989ef80d30 starting on adding some stats to the cutter 5 years ago
Christopher Usher fec8cff185
Merge branch 'master' into chrusher-database-replication 5 years ago
Christopher Usher 027c2900e2 fixes in response to ekim's comments 5 years ago
Christopher Usher 23e3cfce20 Added editor, edit_time and upload_time to thrimshim and cutter updates of the 
database
 5 years ago
Christopher Usher 1dbe585837 retry database connection if it fails 5 years ago
Mike Lang e4d3e418c8 transcode checker: longer retry while waiting for videos to finish 
but still check db often.
This prevents us from using too much api quota on these checks,
while still letting us spot new videos quickly.
 6 years ago
Mike Lang 1f15900b6f cutter: At least for now, don't auto-retry errors 
This leads to rapidly exhausting our upload limit since even a fast failed request
costs the same amount of usage quota as a 1-hour long video.
 6 years ago
Mike Lang fbef4725d7 cutter: Handle case where we are told to stop while looking for candidates 
Previously, it would return None and things would break. Now the None is handled
correctly, and is documented.
 6 years ago
Mike Lang 5cec6ec96e cutter: Reconnect after any error that might be a database error 
After certain kinds of DB error (eg. lost conn), we need to make a new conn
to have things work again. To be safe, we just do it after every error where it might
be a problem.
 6 years ago
Mike Lang fea9ff6c1d cutter: Fix dockerfile, which was missing ffmpeg dependency 6 years ago
Mike Lang e048db0d94 cutter: Fix a failure mode where we never recover from a DB conn failure in TranscodeChecker 
Since we never got a new conn after failure, we would just keep erroring with
"connection already closed" errors.

This isn't applicable to the main cutter loops since a DB failure there will restart the process.
 6 years ago
Mike Lang 85c110ccb4 cutter: Fix typo from when we moved to the client model instead of auth headers 6 years ago
Mike Lang 3647d091f8 Move common google api auth functionality into common 
So we can reuse it for google sheets
 6 years ago
Mike Lang 5b2a1ef6b7 cutter: Implement actual cut methods 
Each method is fairly complicated, but is self-contained and can be examined independently.

cut_jobs in particular contains several extra helpers and directs control flow
via some iterators. This is unfortunately nessecary due to the requests interface.
 6 years ago
Mike Lang ae809c696c cutter: Outline of how main cutter run loop works 
This commit only lays out the main loop, showing the high-level flow
and defining shared utilities. This is for clarity.

The actual methods that do the work will be implemented seperately.
 6 years ago
Mike Lang 80c1a66aa0 cutter: Implement TranscodeChecker 
It runs on an interval, fetching all videos in TRANSCODING from the DB,
checking them against youtube, and then updating any that are done.

It should be noted that youtube somewhat lies about what being "done" means,
but this is a better approximation than nothing.
 6 years ago
Mike Lang 3ce8360a1e cutter: Add database manager and connections 
One connection each for transcode checker and cutter.
We don't need more than one each since both workers only ever do one thing at once.
 6 years ago
Mike Lang fdd245a6d9 cutter: Add lightweight youtube client 
Provides basic youtube api calls, and gets passed into both transcode checker and cutter.

The official youtube client library is many orders of magnitude larger and more complicated,
and can't actually do what we want (stream an upload of unknown size).
 6 years ago
Mike Lang e4b6110fd7 cutter: Add initial outline 
The cutter has two jobs:
* To cut videos, taking them through states EDITED -> TRANSCODING
* To monitor TRANSCODING videos for when they're complete

We run these as separate greenlets with their own DB connections,
and if either dies we gracefully shut down the other.
 6 years ago