henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
1,201 Commits
37 Branches
0 Tags
20 MiB
mlang/suspicious-skew
master
mike/encode-worker
mike/playlist-debug
mike/faster-chat-merge
HubbeKing-multiplatform-builds
thrimbletrimmer-solid
bus-progress-ui
bus-post-processing
chrusher/bus_synthesizer
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'e0c48fb989'
${ noResults }
Commit Graph

1201 Commits (e0c48fb9898aaa71b5ae80a0eb63ba22a6a50b04)
 

Author SHA1 Message Date
Mike Lang e0c48fb989 sheetsync: Extract db table name into a variable 
So we can swap it out for playlists later
 12 months ago
Mike Lang 33986cb7d9 sheetsync: Handle sheet_name as an input column to avoid special casing 
sheet_name must always be present on a sheet row, but is only present on a db row
if the db actually stores that data.

As a side benefit, the db will now update if sheet_name changes.
 12 months ago
Mike Lang 3b78322092 sheetsync: Update streamlog middleware for section -> tab rename 12 months ago
Mike Lang 8a7a489c97 sheetsync: Better error handling for running out of space when creating rows 12 months ago
Mike Lang e1020bd90b sheetsync: Namespace all logs and metrics behind a sheetsync "name" 
This helps differentiate the multiple syncs we now have and will have:
- syncing events from streamlog
- reverse syncing events to sheets
- syncing playlists
 12 months ago
Mike Lang 35f5d3d3d9 sheetsync: Rename "row" and "event" to "sheet_row" and "db_row" 
First step to combining event and playlist sync into one codepath.

No actual behaviour changes intended.
 12 months ago
Mike Lang c0420f960f Get sheets working again 12 months ago
Mike Lang f4672aad8c Don't make sheet name an input column, go back to special casing it on row create 
This means it won't update if put in the wrong place,
but avoids issues with reverse sync trying to write it out when it's not an actual column
 12 months ago
Mike Lang 1c935765cc error is always a string, it just might be empty 12 months ago
Mike Lang 54e0ce59a5 wip: 12 months ago
Mike Lang 2854728e75 clear up confusion with empty string vs None 12 months ago
Mike Lang 9038c3a5a7 wip: fixes 12 months ago
Mike Lang eaff77f6b0 sheetsync: pass in event id instead of event name 12 months ago
Mike Lang cc010b210a Fix streamlog formatting 12 months ago
Mike Lang bbb05d9f34 docker-compose: Allow setting up sheetsync for sheets or streamlog 12 months ago
Mike Lang a5745f7bbd sheetsync: Change how options are specified to allow multiple backends / syncs 12 months ago
Mike Lang 045c5057b4 Implement reverse sync mode 
This is a mode where all data flows one-way from the database to the sheet.
It is intended to be used to populate an empty sheet from database events,
possibly sourced from somewhere else.

To make this work, a few changes were required:
* Track which ids we've seen so we know what events were not matched with a row
* Allow `row` to be None in sync_rows
* When it is, call the middleware to create a new row with a new id
* In sheets, this is implemented by tracking the last empty rows we saw, and claiming them as needed.
 12 months ago
Mike Lang 9409943f67 sheetsync: Remove pick_worksheets() from middleware api 
Instead, get_rows() makes that decision internally if needed.
 12 months ago
Mike Lang 61c5cba5d2 sheetsync: Remove worksheet from middleware apis 
since it's now baked into the row dict
 12 months ago
Mike Lang 2d7dcbc64e sheetsync: Always have sheet name as part of row dict 12 months ago
Mike Lang 4ed6b5fc0f bugfix 12 months ago
Mike Lang 3e9f6ec4d1 sheetsync: Use streamlog section instead of deriving day from start time 12 months ago
Mike Lang 931f193fb5 sheetsync: Implement Streamlog middleware 12 months ago
Mike Lang cdf185cafb sheetsync: Move edit url management into Sheets middleware 
As streamlog doesn't require it.
 12 months ago
Mike Lang 190ddd66c8 Change database primary key from UUID to TEXT 
We still store uuids, but in text form.
This allows us to store non-UUID ids for systems that have other ids.
 12 months ago
Mike Lang 5c6cb2900c Sheetsync: Split into the main loop logic + sheets-specific middleware 
NOTE ON CONFLICTS

In master, we moved sheets.py to common as it only contained a generic client.
Now sheets.py also contains specific sheetsync stuff.

Our resolution:
- Keep the generic version in common
- Keep the old version verbatim (including the now-redundant generic client) in sheetsync

We will move the sheetsync implementation to the generic client after the rebase is complete.
 12 months ago
Mike Lang 13a76c335b sheetsync: Split playlist runloop from normal sheets 12 months ago
Mike Lang 9122ba0ae0 sheetsync: Move id allocation out of sync_row() 12 months ago
Mike Lang e505a09081 sheetsync: Move row_index variable into row dict 12 months ago
Mike Lang f6658721b3 Basic streamlog api 12 months ago
Mike Lang 6d3c901433 Revert "sheetsync: Support archive sheet" 
This reverts commit b93597c274.
 12 months ago
Mike Lang d1439e7376 Revert "sheetsync: unmapped columns aren't a problem." 
This reverts commit 5256577d00.
 12 months ago
Mike Lang 9dec3cb729 Allow the youtube-manual upload backend to modify videos 
So we can modify youtube videos we manually uploaded.
If the video isn't actually ours, it will error when the modify is attempted which is ok.
 12 months ago
Mike Lang 49dd25c0fa playlist_manager: Improved handling of playlist outdated errors 
By restarting the update immediately instead of waiting for the next one.
We only try this up to 3 times to prevent excessive quota usage if it keeps happening.
 12 months ago
Mike Lang 90a6b5ee17 playlist_manager: Check for re-orderings without doing them initially 
Previously, we would only do reorderings if we were refreshing the playlist
for some other reason (eg. a video was inserted).

We want to refresh the playlist before attempting reorderings,
so we split the routine into two parts:
- A part that finds out-of-order videos and returns a list of moves to make.
- A part that executes those moves.

We do the former before AND after refreshing, and the latter only with the result from after.
 12 months ago
Mike Lang 0a55a74efd playlist_manager: Reorder first/last videos when needed 12 months ago
Mike Lang ce8e7bb3b4 playlist_manager: Make a specialized error for api requests 
So we can easily check the status code.
 12 months ago
Mike Lang a802cbb1d2 playlist_manager: Sort first/last videos correctly 
In addition to sorting the videos themselves into the correct spots,
we need to special case them when scanning for the correct place to insert other videos.

Note this only places them in the correct place on insert,
which requires they be set in the playlist config BEFORE being inserted.

A follow-up commit will handle the case of needing to re-order them post insert.
 12 months ago
Mike Lang fae5267941 playlist_manager: Fetch first/last event info when fetching playlists 
And switch to passing around a namedtuple of these + tags instead of just tags.

To avoid confusion with the list of videos in the playlist, we refer to this data as the "playlist config".
 12 months ago
Mike Lang 115fd8bba3 schema: Add first and last event ids to playlist table 
These represent a pinned first/last video in a playlist.
On the choice of a video id vs an event id:
- Event ids are known before video ids, so we can "set and forget" before a video is uploaded
- No need to re-set if an event's video is re-edited or changed
- In cases where an external video is desired, we can use manual link to associate an event with it

Since we're referencing a primary key, we might as well also make it a proper foreign key
with sensible delete behaviour, though in practice we never delete events.
 12 months ago
Mike Lang e9281e652c playlist_manager: Cache playlist entry IDs along with video IDs 
This is required in order to be able to move entries later.
Note our view of entry IDs may always be out of date, so any time you use one
you have to handle it no longer existing.
 12 months ago
Mike Lang 8c1a8e717e playlist_manager: Wrap lookups of playlist_state into a function 
This prevents accidents where we do a simple lookup instead of a .get()
 12 months ago
Mike Lang 8799014735 playlist_manager: Convert format() to f-strings 
Just some general cleanup.
 12 months ago
Mike Lang c6c279356c playlist_manager: Explicitly say when a variable is a "playlist_id" vs a "playlist" (a list of videos) 
This makes the code much easier to read by making it clear what each variable actually refers to.
 12 months ago
Mike Lang d9f521107f playlist_manager: Clarify set operations 
Use slightly more verbose opertations with intermediate variables that make it clear what we're doing
 12 months ago
Mike Lang 005783ab87 playlist_manager: Fix typo in comment 12 months ago
Mike Lang 6b0a025812 Add transition support to fast cuts 1 year ago
Mike Lang c8724a1e63 rewrite fast cuts to support transitions being allowed later 
In theory there should be no change in actual output for no-transition cuts,
even though we're handling the logic in a very different way.

This doesn't actually allow transitions, but sets up most of what is needed
 1 year ago
Mike Lang 066d10f94a Full cut: Support video transitions 
We support all preset transitions in the xfade filter,
as well as a handful of "custom" ones we define.

We only support an audio cross-fade. We may want to support J and L audio cuts (switch audio
before/after the transition) later.
 1 year ago
Mike Lang 5fbdaf8422 full cuts: Support multiple ranges 
This allows full cuts to support multiple ranges in the same way fast cuts do,
by using multiple inputs to ffmpeg and concat filters joining them.

This will be easy to add transitions to later as this is "just" replacing a concat filter
with an xfade + afade filter.
 1 year ago