henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
1,380 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '9c6340a5e0'
${ noResults }
Commit Graph

1380 Commits (9c6340a5e04e9f982be2217c5721c4fc8590fe20)
 

Author SHA1 Message Date
Mike Lang 596cd92644 cutter: Add more specific error handling to upload backends 
For youtube, know that 4xx's are safe even if finalizing was set.

For local, make all disk errors retryable since it doesn't matter.
 5 years ago
Mike Lang 736040435c Refactor error handling in uploads 
Instead of handling each error condition seperately,
we raise an UploadError which includes whether it's retryable.

The advantage of this is that upload backends can also raise an UploadError
to indicate two conditions it currently cannot:
	That an error is unretryable
	That an error is retryable, even if the row was already in finalizing

Under this scheme, errors while cutting become unretryable UploadErrors,
and unhandled exceptions in uploading become retryable UploadErrors if
the row is not yet finalizing only.
 5 years ago
Mike Lang e435abf72e
Merge pull request #114 from ekimekim/mike/fixes 
Grab-bag of cutter fixes
 5 years ago
Mike Lang 40458d9d7f local backend: Use original version of title in write_info 
instead of safe version
 5 years ago
Mike Lang 80d829b83b full cut: ffmpeg requires a seekable output file 
Most formats like mp4 require ffmpeg to make changes at the start of the file
throughout writing.

Unfortunately, this prevents us from streaming the upload as we cut it.

Instead, we spool to a temporary file until ffmpeg exits,
then upload that all at once.
 5 years ago
Christopher Usher f4cd3f546e removed comments no longer needed 5 years ago
Christopher Usher 51e4520826 replaced warnings.warn with logger.warn 5 years ago
Christopher Usher 7d85eb7272 warn about and ignore files that don't parse as segments 5 years ago
Mike Lang 3a9543a4b5 Suppress less ffmpeg output when cutting 
The "fatal" level was causing some useful errors to be suppressed.
 5 years ago
Mike Lang 3fbbe59b00 cutter Local backend: Fix typo and file extension when full cutting 5 years ago
Mike Lang b0a71fd9f1 cutter: Fix mistake in check_candidate error handling 5 years ago
Mike Lang ba53172cbc cutter: Get video link from backend instead of hard-coding youtube 5 years ago
Mike Lang 12decf015e Fix multiple typos and mistakes with full cuts 5 years ago
Mike Lang ed2395d08e cutter: Fix typo that prevented backends from being configured 5 years ago
Mike Lang bf55b1c75e
Merge pull request #103 from ekimekim/mike/cutter/full-cut 
Resurrect non-experimental cut, now dubbed "full" (vs "fast") cut
 5 years ago
Mike Lang c970677a76
full cut: Fix a typo 5 years ago
Mike Lang 1218b694dd cutter: Allow fast or full cuts per location 
On full cut, use encoding settings determined by the backend.
 5 years ago
Mike Lang 09887f17ab restreamer: Add option to download full cut instead of fast cut 
Transcoding to mp4 with no specific encoding args as a reasonable default.
 5 years ago
Mike Lang d3e1d6b4fc Resurrect non-experimental cut, now dubbed "full" (vs "fast") cut 
In a fast cut, we edit the first and last segments then concatenate them all.
However, this leads to some tiny but perciptible artifacting around the border
of the first and second (and second-last and last) segments.

A full cut is much slower, but re-encodes the video into the desired format
and is more reliable.

We want both options to be available.

With this commit, we only add the option, we don't use it in restreamer or cutter.
 5 years ago
Mike Lang 6533eed4d4
Merge pull request #102 from ekimekim/mike/cutter/tags-and-things 
tags, other video presentation tweaks and things
 5 years ago
Mike Lang 40c4baef0f youtube upload: Set category and language settings 
configured on a per-location basis.
 5 years ago
Mike Lang 5e0d5b9ddc cutter: Add category and sheet_name as video tags 
This gives us tags on categories, and on which day's sheet the event came from.

We use these to make automatic playlists.
 5 years ago
Mike Lang cbd0ef3d9e cutter: Add title header, description footer and static tags 
These are pre-canned parts of the video metadata that we want to be configurable.
 5 years ago
Mike Lang 48593e2b06 database, sheetsync: Add worksheet name column 'sheet_name' 
This tells us which sheet a row came from
(so we don't need to scan every sheet to find it if we're trying to do
lookups in that direction).

It is also needed in order to tag the videos with the Day number.
 5 years ago
Mike Lang 1159a518f0
Merge pull request #100 from ekimekim/mike/cutter/multiple-locations 
cutter: Allow multiple upload locations
 5 years ago
Mike Lang 1a177931a6 cutter: Simplify upload backend interface 
Only support iterable of string, not file-like or string.

This is a minor usability loss but we only call this from one place anyway
and it's always an iterable of string.
 5 years ago
Mike Lang 2194e47ab5
Merge pull request #101 from ekimekim/mike/cutter/set-error-on-holes 
cutter: Set an error message to warn humans when it discards a candidate due to holes
 5 years ago
Mike Lang f8039028c4 local upload backend: Add a testing option to show info upload_video() was called with 
This is useful because it lets us test uploads and tag logic without involving youtube.
 5 years ago
Mike Lang d6ed138db8 cutter: Set an error message to warn humans when it discards a candidate due to holes 
This prevents videos being stuck in EDITED with no visible problem when
they contain holes, but is likely to false positive sometimes.

This is fine though, as it's just a human-readable warning and
it will be cleared as soon as any node accepts the row to be cut.
 5 years ago
Mike Lang 75acd978de cutter: Implement a Local upload location 
This cuts to local disk, useful for self-hosting and testing.
 5 years ago
Mike Lang 8e6c40610a cutter: Allow config to manually disable transcode checking for location 
This deals with the problem where multiple youtube locations that refer
to the same actual account (but with different settings) will all try to check
for when videos are done transcoding, when only one is needed.
 5 years ago
Mike Lang ecf58dfbfc cutter: Add ability to configure multiple upload locations 
Cutter now takes a 'config' arg which is a json blob with detail
on each upload location. This is a bit nasty if you're trying to run it manually
but was the easiest way to transfer the config data from docker-compose.jsonnet
to the actual application.
 5 years ago
Mike Lang 514a7a4d08 Split cutter and sheetsync creds so they can be different 
This also means that they can't access each others' creds,
which is good for security.
 5 years ago
Mike Lang 32d352e2ba cutter: Set video directly to DONE if upload backend doesn't need transcoding 
This will be the case for probably all non-youtube backends.
 5 years ago
Mike Lang 39f006fdab cutter: Replace youtube client with generic upload backend and specific youtube implementaton 
Still to go: Actually constructing the correct backend based on given config,
switching behaviour based on needs_transcode.
 5 years ago
Mike Lang 15f879682d cutter: Allow job to specify upload location 
This lays the groundwork for being able to cut to many upload locations.
Right now, only a single location can be configured, and only youtube is supported.
 5 years ago
Christopher Usher 9ed8facce8 fixed bug in error handling 5 years ago
Christopher Usher a63e18c0a9 added state to upload_errors 5 years ago
Christopher Usher 228468749c changes based on ekim's suggestions 5 years ago
Christopher Usher 5c08540c0a stats for the cutter 5 years ago
Christopher Usher 989ef80d30 starting on adding some stats to the cutter 5 years ago
Christopher Usher f9ce41ef32 metrics for the sheetsync 5 years ago
Christopher Usher 928f9733d2 horrible bug with negative times fixed 5 years ago
Christopher Usher a2e47b98f5 fixes -- in dates and the lack of the preshow 5 years ago
Christopher Usher ccb7f3c684 now use parse_segment_path to get hash from filename 5 years ago
Christopher Usher a6303c38ce fixed parse_segment_path to allow just a filename to be parsed 5 years ago
Christopher Usher 36da1926d0 fixes for ekims suggestions 5 years ago
Christopher Usher 34785d1179 now checking the hashes 5 years ago
Christopher Usher b0562495d2 reject mismatched hashes; more metrics 5 years ago
Christopher Usher 120a5a7de0 started on checking the hash 5 years ago