Commit Graph

130 Commits (master)

Author SHA1 Message Date
Mike Lang d6ed138db8 cutter: Set an error message to warn humans when it discards a candidate due to holes
This prevents videos being stuck in EDITED with no visible problem when
they contain holes, but is likely to false positive sometimes.

This is fine though, as it's just a human-readable warning and
it will be cleared as soon as any node accepts the row to be cut.
5 years ago
Mike Lang 75acd978de cutter: Implement a Local upload location
This cuts to local disk, useful for self-hosting and testing.
5 years ago
Mike Lang 8e6c40610a cutter: Allow config to manually disable transcode checking for location
This deals with the problem where multiple youtube locations that refer
to the same actual account (but with different settings) will all try to check
for when videos are done transcoding, when only one is needed.
5 years ago
Mike Lang ecf58dfbfc cutter: Add ability to configure multiple upload locations
Cutter now takes a 'config' arg which is a json blob with detail
on each upload location. This is a bit nasty if you're trying to run it manually
but was the easiest way to transfer the config data from docker-compose.jsonnet
to the actual application.
5 years ago
Mike Lang 32d352e2ba cutter: Set video directly to DONE if upload backend doesn't need transcoding
This will be the case for probably all non-youtube backends.
5 years ago
Mike Lang 39f006fdab cutter: Replace youtube client with generic upload backend and specific youtube implementaton
Still to go: Actually constructing the correct backend based on given config,
switching behaviour based on needs_transcode.
5 years ago
Mike Lang 15f879682d cutter: Allow job to specify upload location
This lays the groundwork for being able to cut to many upload locations.
Right now, only a single location can be configured, and only youtube is supported.
5 years ago
Christopher Usher 9ed8facce8 fixed bug in error handling 5 years ago
Christopher Usher a63e18c0a9 added state to upload_errors 5 years ago
Christopher Usher 228468749c changes based on ekim's suggestions 5 years ago
Christopher Usher 5c08540c0a stats for the cutter 5 years ago
Christopher Usher 989ef80d30 starting on adding some stats to the cutter 5 years ago
Christopher Usher fec8cff185
Merge branch 'master' into chrusher-database-replication 5 years ago
Christopher Usher 027c2900e2 fixes in response to ekim's comments 5 years ago
Christopher Usher 23e3cfce20 Added editor, edit_time and upload_time to thrimshim and cutter updates of the
database
5 years ago
Christopher Usher 1dbe585837 retry database connection if it fails 5 years ago
Mike Lang e4d3e418c8 transcode checker: longer retry while waiting for videos to finish
but still check db often.
This prevents us from using too much api quota on these checks,
while still letting us spot new videos quickly.
5 years ago
Mike Lang 1f15900b6f cutter: At least for now, don't auto-retry errors
This leads to rapidly exhausting our upload limit since even a fast failed request
costs the same amount of usage quota as a 1-hour long video.
5 years ago
Mike Lang fbef4725d7 cutter: Handle case where we are told to stop while looking for candidates
Previously, it would return None and things would break. Now the None is handled
correctly, and is documented.
5 years ago
Mike Lang 5cec6ec96e cutter: Reconnect after any error that might be a database error
After certain kinds of DB error (eg. lost conn), we need to make a new conn
to have things work again. To be safe, we just do it after every error where it might
be a problem.
5 years ago
Mike Lang fea9ff6c1d cutter: Fix dockerfile, which was missing ffmpeg dependency 5 years ago
Mike Lang e048db0d94 cutter: Fix a failure mode where we never recover from a DB conn failure in TranscodeChecker
Since we never got a new conn after failure, we would just keep erroring with
"connection already closed" errors.

This isn't applicable to the main cutter loops since a DB failure there will restart the process.
6 years ago
Mike Lang 85c110ccb4 cutter: Fix typo from when we moved to the client model instead of auth headers 6 years ago
Mike Lang 3647d091f8 Move common google api auth functionality into common
So we can reuse it for google sheets
6 years ago
Mike Lang 5b2a1ef6b7 cutter: Implement actual cut methods
Each method is fairly complicated, but is self-contained and can be examined independently.

cut_jobs in particular contains several extra helpers and directs control flow
via some iterators. This is unfortunately nessecary due to the requests interface.
6 years ago
Mike Lang ae809c696c cutter: Outline of how main cutter run loop works
This commit only lays out the main loop, showing the high-level flow
and defining shared utilities. This is for clarity.

The actual methods that do the work will be implemented seperately.
6 years ago
Mike Lang 80c1a66aa0 cutter: Implement TranscodeChecker
It runs on an interval, fetching all videos in TRANSCODING from the DB,
checking them against youtube, and then updating any that are done.

It should be noted that youtube somewhat lies about what being "done" means,
but this is a better approximation than nothing.
6 years ago
Mike Lang 3ce8360a1e cutter: Add database manager and connections
One connection each for transcode checker and cutter.
We don't need more than one each since both workers only ever do one thing at once.
6 years ago
Mike Lang fdd245a6d9 cutter: Add lightweight youtube client
Provides basic youtube api calls, and gets passed into both transcode checker and cutter.

The official youtube client library is many orders of magnitude larger and more complicated,
and can't actually do what we want (stream an upload of unknown size).
6 years ago
Mike Lang e4b6110fd7 cutter: Add initial outline
The cutter has two jobs:
* To cut videos, taking them through states EDITED -> TRANSCODING
* To monitor TRANSCODING videos for when they're complete

We run these as separate greenlets with their own DB connections,
and if either dies we gracefully shut down the other.
6 years ago