henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
1,218 Commits
37 Branches
0 Tags
20 MiB
mike/data-format-docs
master
mike/encode-worker
mike/playlist-debug
mike/faster-chat-merge
HubbeKing-multiplatform-builds
thrimbletrimmer-solid
bus-progress-ui
bus-post-processing
chrusher/bus_synthesizer
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'a0d54e8c26'
${ noResults }
Commit Graph

1218 Commits (a0d54e8c26268030dce75adec4d37047d18f10bd)
 

Author SHA1 Message Date
Mike Lang a0d54e8c26 Support archive sync in sheetsync again 1 year ago
Mike Lang fcc93ae2c4 Restructure playlists table 
to have all tags, not just playlists.
Notably, this means playlist ids may be null.
 1 year ago
Mike Lang e08cd82325 Fix some typos in StreamLogPlaylistsMiddleware 1 year ago
Mike Lang cda5eb233a Fix: Move events-specific observe_rows() out of SheetSync 1 year ago
Mike Lang 6b462275a9 schema: Playlist tags may be null, indicating "match nothing" 
This is intended for partially-entered playlist data where tags have not been specified yet.
 1 year ago
Mike Lang 3007bf91a4 Update sheetsync CLI interface to allow any combination of types and backends 1 year ago
Mike Lang 65cb58db4f sheetsync: Replace old special-case PlaylistSync with SheetSync subclass 1 year ago
Mike Lang e26949c078 Fix sheetsync to use Sheets client from common 1 year ago
Mike Lang 5b4769cf90 implement SheetsPlaylistMiddleware and fix lots of SheetsMiddleware issues 1 year ago
Mike Lang 909e198eff sheetsync: Streamlog middleware for playlists 1 year ago
Mike Lang 176e37cccb sheetsync: Make the middleware interface explicit with a base class 
The main purpose here is just documentation, so it's clear what the semantics of a middleware is.
 1 year ago
Mike Lang 83ff3c69c9 sheetsync: Split SheetsMiddleware into a common base class and events specific 1 year ago
Mike Lang b3713362d0 sheetsync: Only consider a row missing if we did a full sync 
This prevents things being spuriously missing because they were in a worksheet we didn't sync.
 1 year ago
Mike Lang 95ca38fe1c Split SheetSync object to be fully generic and have an EventsSync object for events 1 year ago
Mike Lang da33235f20 sheetsync: Move extra metrics columns into a variable 1 year ago
Mike Lang 7b4ca69acf sheetsync: Move various config fields to be defined on the class object 
Also make the columns lists into sets
 1 year ago
Mike Lang 86dc30b52c sheetsync: pull metric gathering into a seperate function 1 year ago
Mike Lang e0c48fb989 sheetsync: Extract db table name into a variable 
So we can swap it out for playlists later
 1 year ago
Mike Lang 33986cb7d9 sheetsync: Handle sheet_name as an input column to avoid special casing 
sheet_name must always be present on a sheet row, but is only present on a db row
if the db actually stores that data.

As a side benefit, the db will now update if sheet_name changes.
 1 year ago
Mike Lang 3b78322092 sheetsync: Update streamlog middleware for section -> tab rename 1 year ago
Mike Lang 8a7a489c97 sheetsync: Better error handling for running out of space when creating rows 1 year ago
Mike Lang e1020bd90b sheetsync: Namespace all logs and metrics behind a sheetsync "name" 
This helps differentiate the multiple syncs we now have and will have:
- syncing events from streamlog
- reverse syncing events to sheets
- syncing playlists
 1 year ago
Mike Lang 35f5d3d3d9 sheetsync: Rename "row" and "event" to "sheet_row" and "db_row" 
First step to combining event and playlist sync into one codepath.

No actual behaviour changes intended.
 1 year ago
Mike Lang c0420f960f Get sheets working again 1 year ago
Mike Lang f4672aad8c Don't make sheet name an input column, go back to special casing it on row create 
This means it won't update if put in the wrong place,
but avoids issues with reverse sync trying to write it out when it's not an actual column
 1 year ago
Mike Lang 1c935765cc error is always a string, it just might be empty 1 year ago
Mike Lang 54e0ce59a5 wip: 1 year ago
Mike Lang 2854728e75 clear up confusion with empty string vs None 1 year ago
Mike Lang 9038c3a5a7 wip: fixes 1 year ago
Mike Lang eaff77f6b0 sheetsync: pass in event id instead of event name 1 year ago
Mike Lang cc010b210a Fix streamlog formatting 1 year ago
Mike Lang bbb05d9f34 docker-compose: Allow setting up sheetsync for sheets or streamlog 1 year ago
Mike Lang a5745f7bbd sheetsync: Change how options are specified to allow multiple backends / syncs 1 year ago
Mike Lang 045c5057b4 Implement reverse sync mode 
This is a mode where all data flows one-way from the database to the sheet.
It is intended to be used to populate an empty sheet from database events,
possibly sourced from somewhere else.

To make this work, a few changes were required:
* Track which ids we've seen so we know what events were not matched with a row
* Allow `row` to be None in sync_rows
* When it is, call the middleware to create a new row with a new id
* In sheets, this is implemented by tracking the last empty rows we saw, and claiming them as needed.
 1 year ago
Mike Lang 9409943f67 sheetsync: Remove pick_worksheets() from middleware api 
Instead, get_rows() makes that decision internally if needed.
 1 year ago
Mike Lang 61c5cba5d2 sheetsync: Remove worksheet from middleware apis 
since it's now baked into the row dict
 1 year ago
Mike Lang 2d7dcbc64e sheetsync: Always have sheet name as part of row dict 1 year ago
Mike Lang 4ed6b5fc0f bugfix 1 year ago
Mike Lang 3e9f6ec4d1 sheetsync: Use streamlog section instead of deriving day from start time 1 year ago
Mike Lang 931f193fb5 sheetsync: Implement Streamlog middleware 1 year ago
Mike Lang cdf185cafb sheetsync: Move edit url management into Sheets middleware 
As streamlog doesn't require it.
 1 year ago
Mike Lang 190ddd66c8 Change database primary key from UUID to TEXT 
We still store uuids, but in text form.
This allows us to store non-UUID ids for systems that have other ids.
 1 year ago
Mike Lang 5c6cb2900c Sheetsync: Split into the main loop logic + sheets-specific middleware 
NOTE ON CONFLICTS

In master, we moved sheets.py to common as it only contained a generic client.
Now sheets.py also contains specific sheetsync stuff.

Our resolution:
- Keep the generic version in common
- Keep the old version verbatim (including the now-redundant generic client) in sheetsync

We will move the sheetsync implementation to the generic client after the rebase is complete.
 1 year ago
Mike Lang 13a76c335b sheetsync: Split playlist runloop from normal sheets 1 year ago
Mike Lang 9122ba0ae0 sheetsync: Move id allocation out of sync_row() 1 year ago
Mike Lang e505a09081 sheetsync: Move row_index variable into row dict 1 year ago
Mike Lang f6658721b3 Basic streamlog api 1 year ago
Mike Lang 6d3c901433 Revert "sheetsync: Support archive sheet" 
This reverts commit b93597c274.
 1 year ago
Mike Lang d1439e7376 Revert "sheetsync: unmapped columns aren't a problem." 
This reverts commit 5256577d00.
 1 year ago
Mike Lang 9dec3cb729 Allow the youtube-manual upload backend to modify videos 
So we can modify youtube videos we manually uploaded.
If the video isn't actually ours, it will error when the modify is attempted which is ok.
 1 year ago