henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
988 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '791776d913'
${ noResults }
Commit Graph

96 Commits (791776d91313095f8a79d785d3a658e7a868498c)

Author SHA1 Message Date
Mike Lang 39b48c8b3a Don't attempt to upload a video with a >2MB thumbnail 2 years ago
Mike Lang b155c53b74 cutter: Correctly set video id and link on failed thumbnail upload 2 years ago
Mike Lang 2fee3a6e9d cutter: Don't set error on concurrent video update 
If two cutters modify a video at the same time, youtube may respond with a 409 to one of them.
We want to treat that as a retryable error, and ideally let another cutter finish it instead.
 2 years ago
Mike Lang 8b8b2a347e Fix typo in respecting upload location 2 years ago
Mike Lang 7fc9d3d2df cutter: Transcode and Update checkers should only affect videos for that upload location 2 years ago
Mike Lang ab985cf1b0 Fix syntax error 2 years ago
Mike Lang cfe93e31dd Fix a bug where modified videos are missing static tags 2 years ago
Mike Lang bd519369bd thumbnail fixes 2 years ago
Mike Lang 31c3f0b8b1 OS-level dependencies for Pillow 2 years ago
Mike Lang 45c46df8bb Add thumbnail templating code 2 years ago
Mike Lang d3e21ae9b0 Implement thumbnails in cutter 2 years ago
Mike Lang 46468409cb Add "public" column to control if videos are unlisted 
This column is modifyable.
This replaces the old "hidden" argument to youtube upload backends.
 2 years ago
Mike Lang c4a1d72240 Add new state to postgres and implement cutter logic 2 years ago
Mike Lang 101468cbca Fix a bug when reporting errors claiming candidates 
The SQL args were the wrong way around, so we tried to set error to the id.
Thankfully this failed as the id queried is not a valid uuid.

Fix by making args named like every other usage of query()
 3 years ago
Mike Lang 5e60344522 cutter: Fix errors due to unsupported cut type not being handled correctly 
Previously, determining the correct cut type and setting up the cut iterator
wasn't an operation we expected to be able to fail.
It happened outside all error handling blocks.

However, due to the multi-range work, this can now fail if we are requesting
an unsupported combination of transitions and cut type.

In order to correctly handle this like other cut errors, ie. set to UNEDITED
and set error column, we move this logic into the upload_wrapper right before we iterate
through the resulting cut.
 3 years ago
Mike Lang a47c29fff4 Link images to github repo by adding a LABEL 
When pushed, this tells github to associate the ghcr.io repo that was pushed to
with the github repo specified (the owner needs to match).

This does a few things.
Most importantly, this automatically gives github actions credentials to push to these
repositories when run in the context of the wubloader repo.
 3 years ago
Mike Lang aab8cf2f0f Set up plumbing for multi-range videos and implement no-transition fast cut videos only 
This is the simplest case as we can just cut each range like we already do,
then concat the results.

We still allow for the full design in the database and cutter, but error out if transitions
is ever anything but hard cuts or if it's a full cut.

We also update the restreamer to allow accepting ranges, however for usability we still allow
the old "just one start and end" args.

Note this changes the thrimshim API to give and take the new "video_ranges" and "video_transitions" columns.
 3 years ago
Mike Lang 62bd6539ea Unpin gevent as that was a workaround for a py2 issue 3 years ago
Mike Lang 21856c68aa Fix all instances of file.write() for py3 
In python 3, file.write() may do a partial write and returns the number of characters written.
In order to not lose data, we need to wrap every instance of file.write() with our new
common.writeall() wrapper that loops until the data is actually written.
 3 years ago
Mike Lang a56f6859bb more py3 fixes 3 years ago
Mike Lang f2a8007bf7 Fix build dependency issues 3 years ago
Mike Lang 50231a5154 py3 fixes for cutter 3 years ago
HubbeKing 6d790a1b36 Do a first naive pass for py3 compatibility 
Check that open() calls for reading and writing use binary modes
Use alpine version with py3-pip package
Use python3 in Dockerfile CMD
Remove sys.setdefaultencoding() "hack"
Simplify ensure_directory() in common.common package
 3 years ago
Mike Lang f0546e2ee3 Pin gevent to 1.5a2 to avoid https://github.com/gevent/gevent/issues/1711 3 years ago
Mike Lang b53fcd65a0 Add dependencies required to install psycopg2 from source 
We can't install the binaries as they don't support musl
 4 years ago
HubbeKing 86f7823348 Replace calls to gevent.signal() with gevent.signal_handler() 
gevent.signal() was removed in gevent 1.5a4, see http://www.gevent.org/api/gevent.signal.html
Removed on Feb 5th, see https://github.com/gevent/gevent/pull/1530
 4 years ago
Mike Lang a53786dc2d Add file and make as build dependencies 
gevent now requires these to build. I'm not sure when this changed.
 4 years ago
Mike Lang b9cd76b1a2 Add non-static implict tags in sheetsync 
In order for the upcoming playlist manager to be able to use the DB `tags` column to know
what tags a video has, all the tags it needs need to be present.

Previously, this was a problem because the day and category tags only get added at the cutter
and so wouldn't be listed.

This moves them so they are added when parsing the row in sheetsync.
It also adds the poster moment tag if poster moment is checked.

Note that fully static tags that go on all videos are still only added in cutter,
but the playlist manager doesn't need to care about those (since by definition
they will match every video).
 4 years ago
Mike Lang a30d595fee cutter: Use video's custom tags when uploading 4 years ago
Mike Lang 4a2fe7a6ed cutter: Explicitly set mime type of uploads correctly 5 years ago
Mike Lang fde2758275 cutter: Fix bug where uploader was cleared on non-retryable error 
Instead of on retryable error
 5 years ago
Mike Lang a002619c4c youtube upload: Explicitly set mime type 5 years ago
Mike Lang a438d86f80 cutter: Fix multiple problems with logging errors 5 years ago
Mike Lang 4bbcc8bc06 Revert "Merge pull request #155 from ekimekim/mike/manual-uploads" 
This reverts commit 99de586353, reversing
changes made to 4b04f70b6f.

We don't need this feature and it complicates things and adds bugs.
 5 years ago
Mike Lang c580671da2 Create upload pending state 5 years ago
Mike Lang 605fedeb52 cutter: Log when upload errors occur 
This got lost in the shuffle at some point.
 5 years ago
Mike Lang e4706232ad cutter: Fix bug in handling cancellation before finalize 
it would always think it's finalized
 5 years ago
Mike Lang 4c0e20983c Local upload backend: Always use .ts extension 
Now that we're doing that for both fast and full cuts.
 5 years ago
Mike Lang 1cb4d41334 cutter: Set upload time when marking video as done without transcoding 5 years ago
Mike Lang a7ac264201 cutter: Fix local upload backend, which was returning video id as a uuid 
instead of a string
 5 years ago
Mike Lang da58021c6f cutter: Fix typo where encoding settings had wrong name 5 years ago
Mike Lang 1ef15780df cutter: Fix lack of info in error handling 5 years ago
Mike Lang c1cb7966af cutter: Default to full cuts, not fast 5 years ago
Mike Lang b2a07ef114
Merge pull request #140 from ekimekim/mike/build-improvements 
Refactor dockerfiles for more shared layers
 5 years ago
Mike Lang 731ef9e2d0 Refactor dockerfiles for more shared layers 
By carefully ensuring most of our dockerfiles are identical in their first few layers,
we only need to build those layers once instead of every time.

In particular, we move installing gevent to before installing common,
so that even when common changes gevent doesn't need to be reinstalled.

This is important because gevent takes ages to install.

Also fixes segment_coverage, which wasn't being installed.
 5 years ago
Mike Lang 3dfe0e8722 Allow thrimshim to safely cancel a job while it is cutting 
This differs from the existing reset row by only suceeding if the upload is not
in finalizing.

We also make some changes to cutter to handle this situation gracefully.
 5 years ago
Mike Lang 4d52b18b04 cutter,restreamer: Set stream=True for full cuts when appropriate 
And also default to a new ffmpeg encoding setting for high-quality mpegts
(ie. still streamable) that is encoded very quickly.
 5 years ago
Mike Lang f9b48bc70e cutter: Add more metrics 5 years ago
Mike Lang 52e6c4ad41 sheetsync, cutter: Collect metrics on http calls 
In particular, to google apis.
 5 years ago
Mike Lang 17af1c4e89 cutter, sheetsync: Wait for DB to connect on startup 
This is a nicer error than crashing in the depths of some error handler
(which is what will happen if the DB goes unavailable while they're running),
and it's a far more common case (eg. the DB is misconfigured) than having it fail
halfway through.

Neither of these services can do anything meaningful without the DB,
so crashing without it is acceptable behaviour.
 5 years ago