henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
848 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'mike/debug-playlists'
${ noResults }
Commit Graph

83 Commits (mike/debug-playlists)

Author SHA1 Message Date
Mike Lang 7649a4e840 Improve WSGIServer graceful shutdown handling 
Previously both restreamer and thrimshim had some complex logic for dealing with
graceful shutdown, in different ways, that was still prone to race conditions.

We replace this with a common method that does it properly.

Fixes #226
 3 years ago
Mike Lang a47c29fff4 Link images to github repo by adding a LABEL 
When pushed, this tells github to associate the ghcr.io repo that was pushed to
with the github repo specified (the owner needs to match).

This does a few things.
Most importantly, this automatically gives github actions credentials to push to these
repositories when run in the context of the wubloader repo.
 3 years ago
Mike Lang 192a0869f7 thrimshim: Fix bug in serializing columns 
Converting all datetime values to string no longer works because video_range
isn't a datetime but a list of datetimes. We switch to a more robust approach
of using the json "default" arg to specify how to serialize.
 3 years ago
Mike Lang aab8cf2f0f Set up plumbing for multi-range videos and implement no-transition fast cut videos only 
This is the simplest case as we can just cut each range like we already do,
then concat the results.

We still allow for the full design in the database and cutter, but error out if transitions
is ever anything but hard cuts or if it's a full cut.

We also update the restreamer to allow accepting ranges, however for usability we still allow
the old "just one start and end" args.

Note this changes the thrimshim API to give and take the new "video_ranges" and "video_transitions" columns.
 3 years ago
Mike Lang 2865088692 thrimshim: Further validation of titles/descriptions 
Youtube doesn't allow < or > characters, or extremely long descriptions.

Fixes #215, #219
 3 years ago
Mike Lang 62bd6539ea Unpin gevent as that was a workaround for a py2 issue 3 years ago
Mike Lang f2a8007bf7 Fix build dependency issues 3 years ago
Mike Lang d42d850153 thrimshim: misc cleanup 3 years ago
HubbeKing 6d790a1b36 Do a first naive pass for py3 compatibility 
Check that open() calls for reading and writing use binary modes
Use alpine version with py3-pip package
Use python3 in Dockerfile CMD
Remove sys.setdefaultencoding() "hack"
Simplify ensure_directory() in common.common package
 3 years ago
Mike Lang f0546e2ee3 Pin gevent to 1.5a2 to avoid https://github.com/gevent/gevent/issues/1711 3 years ago
Mike Lang f8c877775d Add an alternate manual upload mode specifically for youtube 
Adds a built-in "youtube-manual" location which is like "manual" except that it only works
with youtube URLs and populates the video_id column.

The intent is so that we can have playlist_manager manage videos we upload manually,
while still being able to distinguish between that and other manual links that shouldn't
be included (eg. links to third party youtube videos).

This is set when setting a manual link in thrimbletrimmer with a new checkbox, default off.
 4 years ago
Mike Lang 6d789ad42f thrimshim: Handle None values in comparing columns 4 years ago
Mike Lang cb75953e91 thrimshim: Fix a bug preventing submissions 
When comparing old and new video tags, it errors because it's a list, not string.

We change it to apply the transforms to all tags in the list, and also ignore changes in list ordering.
 4 years ago
Christopher Usher 41d85b8614 Remove unicode 4 years ago
Christopher Usher 84f7ef71c1 More helpful UI if the database has changed plus ignore changes that are 
just white space or capitalisation
 4 years ago
Mike Lang 48ef416dfb restreamer, thrimshim: Allow /metrics/* in addition to /metrics 
This allows the metrics proxying from nginx to work even if the path is not rewritten,
which is hard to do in k8s.
 4 years ago
Christopher Usher c19dce6e4e I forgot to add tags 4 years ago
Christopher Usher 27c6b70976 fixes in response to review 4 years ago
Christopher Usher b21a520d3a Checking for changes in the sheet now working in the thrimbletrimmer 4 years ago
Christopher Usher 1216c8e8a5 thrimshim side of checking for changes seems to be working 4 years ago
Christopher Usher 8889648a58 Check whether sheet columns have changed during editing in thrimshim. 4 years ago
Mike Lang b53fcd65a0 Add dependencies required to install psycopg2 from source 
We can't install the binaries as they don't support musl
 4 years ago
HubbeKing 86f7823348 Replace calls to gevent.signal() with gevent.signal_handler() 
gevent.signal() was removed in gevent 1.5a4, see http://www.gevent.org/api/gevent.signal.html
Removed on Feb 5th, see https://github.com/gevent/gevent/pull/1530
 4 years ago
Mike Lang a53786dc2d Add file and make as build dependencies 
gevent now requires these to build. I'm not sure when this changed.
 4 years ago
Mike Lang 7e0c415f83 thrimshim: Add video tags column as required 4 years ago
Mike Lang 4bbcc8bc06 Revert "Merge pull request #155 from ekimekim/mike/manual-uploads" 
This reverts commit 99de586353, reversing
changes made to 4b04f70b6f.

We don't need this feature and it complicates things and adds bugs.
 5 years ago
Mike Lang 9ccc7e4e8d thrimshim: Allow manual_link to set things from UPLOAD_PENDING to TRANSCODING 5 years ago
Mike Lang b2a07ef114
Merge pull request #140 from ekimekim/mike/build-improvements 
Refactor dockerfiles for more shared layers
 5 years ago
Mike Lang 731ef9e2d0 Refactor dockerfiles for more shared layers 
By carefully ensuring most of our dockerfiles are identical in their first few layers,
we only need to build those layers once instead of every time.

In particular, we move installing gevent to before installing common,
so that even when common changes gevent doesn't need to be reinstalled.

This is important because gevent takes ages to install.

Also fixes segment_coverage, which wasn't being installed.
 5 years ago
Mike Lang 3dfe0e8722 Allow thrimshim to safely cancel a job while it is cutting 
This differs from the existing reset row by only suceeding if the upload is not
in finalizing.

We also make some changes to cutter to handle this situation gracefully.
 5 years ago
Mike Lang a42e7b48f6 thrimshim: Allow degraded operation even if DB broken 
Any endpoints that don't need a DB conn will still work fine.

Notably, this includes /defaults, which is needed for
thrimbletrimmer to work in a non-specific-row mode.
 5 years ago
Mike Lang f7f07a2688 Add /defaults to thrimshim, use it to populate config stuff in thrimbletrimmer 
Thrimbletrimmer needs to know stuff like the bustime start time,
default channel, even if it's not looking at a specific row.
 5 years ago
Mike Lang da3cc24ed1 Pass a list of upload locations from thrimshim to thrimbletrimmer 
with the first one being the default.
 5 years ago
Mike Lang b11fe39371 thrimshim: Validate title length, non-empty title/description 
and pass the title prefix / max length through to thrimbletrimmer
 5 years ago
Mike Lang ba746ff6e6 Add title and description header/footer in thrimshim instead of cutter 
This accomplishes two things:
1. It allows thrimshim to properly validate length restrictions (not implemented yet)
2. It means that the database has a record of the values actually written for each of these rows,
instead of that information depending on how the cutter was configured at the time.
 5 years ago
mg 936a2387c8 Change to how default channel is handled. 5 years ago
mg ca11e59511 Adding channel/start time info to Thrimshim to pass to Thrimbletrimmer. 5 years ago
mg 6edd224395 Merging Thrimbletrimmer into Wubloader with latest changes. 5 years ago
Christopher Usher fec8cff185
Merge branch 'master' into chrusher-database-replication 5 years ago
Christopher Usher 027c2900e2 fixes in response to ekim's comments 5 years ago
Christopher Usher 92f6ed5891 function name fix 5 years ago
Christopher Usher 86477fae13 fixes for ekim's comments 5 years ago
Christopher Usher 611a4bd793 added option not to authenticate thrimshim 5 years ago
Christopher Usher 36fe22bf99 authentication implemented and tested as much as possible without authentication in thrimbletrimmer 5 years ago
Christopher Usher 057bd031a2 decorator to avoid duplication authentication code 5 years ago
Christopher Usher 0cf84ef510 code style clean up 5 years ago
mg 64824973ba Adding issuer validation. 5 years ago
mg 96d2566ab2 Adding dependencies and retrieving account info. 5 years ago
Christopher Usher aea0fe1766 Setting up Auth Test function 5 years ago
Christopher Usher d07cb5de44 Placeholder text 5 years ago