henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
556 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '845744cbf6'
${ noResults }
Commit Graph

62 Commits (845744cbf606323ed336ec02e849df8f49377f45)

Author SHA1 Message Date
Mike Lang c580671da2 Create upload pending state 5 years ago
Mike Lang 605fedeb52 cutter: Log when upload errors occur 
This got lost in the shuffle at some point.
 5 years ago
Mike Lang e4706232ad cutter: Fix bug in handling cancellation before finalize 
it would always think it's finalized
 5 years ago
Mike Lang 4c0e20983c Local upload backend: Always use .ts extension 
Now that we're doing that for both fast and full cuts.
 5 years ago
Mike Lang 1cb4d41334 cutter: Set upload time when marking video as done without transcoding 5 years ago
Mike Lang a7ac264201 cutter: Fix local upload backend, which was returning video id as a uuid 
instead of a string
 5 years ago
Mike Lang da58021c6f cutter: Fix typo where encoding settings had wrong name 5 years ago
Mike Lang 1ef15780df cutter: Fix lack of info in error handling 5 years ago
Mike Lang c1cb7966af cutter: Default to full cuts, not fast 5 years ago
Mike Lang b2a07ef114
Merge pull request #140 from ekimekim/mike/build-improvements 
Refactor dockerfiles for more shared layers
 5 years ago
Mike Lang 731ef9e2d0 Refactor dockerfiles for more shared layers 
By carefully ensuring most of our dockerfiles are identical in their first few layers,
we only need to build those layers once instead of every time.

In particular, we move installing gevent to before installing common,
so that even when common changes gevent doesn't need to be reinstalled.

This is important because gevent takes ages to install.

Also fixes segment_coverage, which wasn't being installed.
 5 years ago
Mike Lang 3dfe0e8722 Allow thrimshim to safely cancel a job while it is cutting 
This differs from the existing reset row by only suceeding if the upload is not
in finalizing.

We also make some changes to cutter to handle this situation gracefully.
 5 years ago
Mike Lang 4d52b18b04 cutter,restreamer: Set stream=True for full cuts when appropriate 
And also default to a new ffmpeg encoding setting for high-quality mpegts
(ie. still streamable) that is encoded very quickly.
 5 years ago
Mike Lang f9b48bc70e cutter: Add more metrics 5 years ago
Mike Lang 52e6c4ad41 sheetsync, cutter: Collect metrics on http calls 
In particular, to google apis.
 5 years ago
Mike Lang 17af1c4e89 cutter, sheetsync: Wait for DB to connect on startup 
This is a nicer error than crashing in the depths of some error handler
(which is what will happen if the DB goes unavailable while they're running),
and it's a far more common case (eg. the DB is misconfigured) than having it fail
halfway through.

Neither of these services can do anything meaningful without the DB,
so crashing without it is acceptable behaviour.
 5 years ago
Mike Lang ba746ff6e6 Add title and description header/footer in thrimshim instead of cutter 
This accomplishes two things:
1. It allows thrimshim to properly validate length restrictions (not implemented yet)
2. It means that the database has a record of the values actually written for each of these rows,
instead of that information depending on how the cutter was configured at the time.
 5 years ago
Mike Lang 596cd92644 cutter: Add more specific error handling to upload backends 
For youtube, know that 4xx's are safe even if finalizing was set.

For local, make all disk errors retryable since it doesn't matter.
 5 years ago
Mike Lang 736040435c Refactor error handling in uploads 
Instead of handling each error condition seperately,
we raise an UploadError which includes whether it's retryable.

The advantage of this is that upload backends can also raise an UploadError
to indicate two conditions it currently cannot:
	That an error is unretryable
	That an error is retryable, even if the row was already in finalizing

Under this scheme, errors while cutting become unretryable UploadErrors,
and unhandled exceptions in uploading become retryable UploadErrors if
the row is not yet finalizing only.
 5 years ago
Mike Lang 40458d9d7f local backend: Use original version of title in write_info 
instead of safe version
 5 years ago
Mike Lang 3fbbe59b00 cutter Local backend: Fix typo and file extension when full cutting 5 years ago
Mike Lang b0a71fd9f1 cutter: Fix mistake in check_candidate error handling 5 years ago
Mike Lang ba53172cbc cutter: Get video link from backend instead of hard-coding youtube 5 years ago
Mike Lang ed2395d08e cutter: Fix typo that prevented backends from being configured 5 years ago
Mike Lang 1218b694dd cutter: Allow fast or full cuts per location 
On full cut, use encoding settings determined by the backend.
 5 years ago
Mike Lang d3e1d6b4fc Resurrect non-experimental cut, now dubbed "full" (vs "fast") cut 
In a fast cut, we edit the first and last segments then concatenate them all.
However, this leads to some tiny but perciptible artifacting around the border
of the first and second (and second-last and last) segments.

A full cut is much slower, but re-encodes the video into the desired format
and is more reliable.

We want both options to be available.

With this commit, we only add the option, we don't use it in restreamer or cutter.
 5 years ago
Mike Lang 40c4baef0f youtube upload: Set category and language settings 
configured on a per-location basis.
 5 years ago
Mike Lang 5e0d5b9ddc cutter: Add category and sheet_name as video tags 
This gives us tags on categories, and on which day's sheet the event came from.

We use these to make automatic playlists.
 5 years ago
Mike Lang cbd0ef3d9e cutter: Add title header, description footer and static tags 
These are pre-canned parts of the video metadata that we want to be configurable.
 5 years ago
Mike Lang 1159a518f0
Merge pull request #100 from ekimekim/mike/cutter/multiple-locations 
cutter: Allow multiple upload locations
 5 years ago
Mike Lang 1a177931a6 cutter: Simplify upload backend interface 
Only support iterable of string, not file-like or string.

This is a minor usability loss but we only call this from one place anyway
and it's always an iterable of string.
 5 years ago
Mike Lang f8039028c4 local upload backend: Add a testing option to show info upload_video() was called with 
This is useful because it lets us test uploads and tag logic without involving youtube.
 5 years ago
Mike Lang d6ed138db8 cutter: Set an error message to warn humans when it discards a candidate due to holes 
This prevents videos being stuck in EDITED with no visible problem when
they contain holes, but is likely to false positive sometimes.

This is fine though, as it's just a human-readable warning and
it will be cleared as soon as any node accepts the row to be cut.
 5 years ago
Mike Lang 75acd978de cutter: Implement a Local upload location 
This cuts to local disk, useful for self-hosting and testing.
 5 years ago
Mike Lang 8e6c40610a cutter: Allow config to manually disable transcode checking for location 
This deals with the problem where multiple youtube locations that refer
to the same actual account (but with different settings) will all try to check
for when videos are done transcoding, when only one is needed.
 5 years ago
Mike Lang ecf58dfbfc cutter: Add ability to configure multiple upload locations 
Cutter now takes a 'config' arg which is a json blob with detail
on each upload location. This is a bit nasty if you're trying to run it manually
but was the easiest way to transfer the config data from docker-compose.jsonnet
to the actual application.
 5 years ago
Mike Lang 32d352e2ba cutter: Set video directly to DONE if upload backend doesn't need transcoding 
This will be the case for probably all non-youtube backends.
 5 years ago
Mike Lang 39f006fdab cutter: Replace youtube client with generic upload backend and specific youtube implementaton 
Still to go: Actually constructing the correct backend based on given config,
switching behaviour based on needs_transcode.
 5 years ago
Mike Lang 15f879682d cutter: Allow job to specify upload location 
This lays the groundwork for being able to cut to many upload locations.
Right now, only a single location can be configured, and only youtube is supported.
 5 years ago
Christopher Usher 9ed8facce8 fixed bug in error handling 5 years ago
Christopher Usher a63e18c0a9 added state to upload_errors 5 years ago
Christopher Usher 228468749c changes based on ekim's suggestions 5 years ago
Christopher Usher 5c08540c0a stats for the cutter 5 years ago
Christopher Usher 989ef80d30 starting on adding some stats to the cutter 5 years ago
Christopher Usher fec8cff185
Merge branch 'master' into chrusher-database-replication 5 years ago
Christopher Usher 027c2900e2 fixes in response to ekim's comments 5 years ago
Christopher Usher 23e3cfce20 Added editor, edit_time and upload_time to thrimshim and cutter updates of the 
database
 5 years ago
Christopher Usher 1dbe585837 retry database connection if it fails 5 years ago
Mike Lang e4d3e418c8 transcode checker: longer retry while waiting for videos to finish 
but still check db often.
This prevents us from using too much api quota on these checks,
while still letting us spot new videos quickly.
 5 years ago
Mike Lang 1f15900b6f cutter: At least for now, don't auto-retry errors 
This leads to rapidly exhausting our upload limit since even a fast failed request
costs the same amount of usage quota as a 1-hour long video.
 5 years ago