henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
1,250 Commits
35 Branches
0 Tags
19 MiB
chrusher-postgres-upgrade
master
HubbeKing-multiplatform-builds
thrimbletrimmer-solid
bus-progress-ui
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '07055e3605'
${ noResults }
Commit Graph

83 Commits (07055e3605fa6a66fdbc7fe37d01debaecf64cf5)

Author SHA1 Message Date
Mike Lang 1a5687204d debugging 6 months ago
Mike Lang 20ee382b7f sheetsync: Fix missing arg 
Just hard-code that the tags sheet is always called "Tags" for now, we can fix later
if it ever needs to change.
 6 months ago
Mike Lang 367e6a7a7a sheetsync fixes 
typos and omissions
 6 months ago
Mike Lang eeffeeed10 sheetsync: Deal with reverse syncing properly when not all events are in the list of worksheets 
This is important because archive events should not be reversed.
We only want to create new rows when the row's intended worksheet is in our list of worksheets we sync.
 6 months ago
Mike Lang 96181fd875 Support archive sync in sheetsync again 6 months ago
Mike Lang a3aaa37bb0 Restructure playlists table 
to have all tags, not just playlists.
Notably, this means playlist ids may be null.
 6 months ago
Mike Lang 0f06f96a7c Fix: Move events-specific observe_rows() out of SheetSync 6 months ago
Mike Lang 73eaac16f3 Update sheetsync CLI interface to allow any combination of types and backends 6 months ago
Mike Lang bb4cc8c668 sheetsync: Replace old special-case PlaylistSync with SheetSync subclass 6 months ago
Mike Lang ef11b69f4d Fix sheetsync to use Sheets client from common 6 months ago
Mike Lang 3ffbefea4f sheetsync: Split SheetsMiddleware into a common base class and events specific 6 months ago
Mike Lang bebce3df4c sheetsync: Only consider a row missing if we did a full sync 
This prevents things being spuriously missing because they were in a worksheet we didn't sync.
 6 months ago
Mike Lang 3cdd8f22ad Split SheetSync object to be fully generic and have an EventsSync object for events 6 months ago
Mike Lang 6a9884ee5b sheetsync: Move extra metrics columns into a variable 6 months ago
Mike Lang 37dd7c3e4f sheetsync: Move various config fields to be defined on the class object 
Also make the columns lists into sets
 6 months ago
Mike Lang 5559dffec6 sheetsync: pull metric gathering into a seperate function 6 months ago
Mike Lang 37f42853ec sheetsync: Extract db table name into a variable 
So we can swap it out for playlists later
 6 months ago
Mike Lang 00d0ecc3bb sheetsync: Handle sheet_name as an input column to avoid special casing 
sheet_name must always be present on a sheet row, but is only present on a db row
if the db actually stores that data.

As a side benefit, the db will now update if sheet_name changes.
 6 months ago
Mike Lang 29ff11457e sheetsync: Namespace all logs and metrics behind a sheetsync "name" 
This helps differentiate the multiple syncs we now have and will have:
- syncing events from streamlog
- reverse syncing events to sheets
- syncing playlists
 6 months ago
Mike Lang 87b504a00a sheetsync: Rename "row" and "event" to "sheet_row" and "db_row" 
First step to combining event and playlist sync into one codepath.

No actual behaviour changes intended.
 6 months ago
Mike Lang 20ee79cdb1 Get sheets working again 6 months ago
Mike Lang f89ab6fa43 Don't make sheet name an input column, go back to special casing it on row create 
This means it won't update if put in the wrong place,
but avoids issues with reverse sync trying to write it out when it's not an actual column
 6 months ago
Mike Lang ee4a68af50 clear up confusion with empty string vs None 6 months ago
Mike Lang 3e873ca5f6 wip: fixes 6 months ago
Mike Lang eebfa5885b sheetsync: pass in event id instead of event name 6 months ago
Mike Lang 986a1db964 sheetsync: Change how options are specified to allow multiple backends / syncs 6 months ago
Mike Lang 74869de89d Implement reverse sync mode 
This is a mode where all data flows one-way from the database to the sheet.
It is intended to be used to populate an empty sheet from database events,
possibly sourced from somewhere else.

To make this work, a few changes were required:
* Track which ids we've seen so we know what events were not matched with a row
* Allow `row` to be None in sync_rows
* When it is, call the middleware to create a new row with a new id
* In sheets, this is implemented by tracking the last empty rows we saw, and claiming them as needed.
 6 months ago
Mike Lang 85de9757f7 sheetsync: Remove pick_worksheets() from middleware api 
Instead, get_rows() makes that decision internally if needed.
 6 months ago
Mike Lang 17463d70fe sheetsync: Remove worksheet from middleware apis 
since it's now baked into the row dict
 6 months ago
Mike Lang eec58f2651 sheetsync: Always have sheet name as part of row dict 6 months ago
Mike Lang fa9a4b70bb bugfix 6 months ago
Mike Lang d064522d60 sheetsync: Move edit url management into Sheets middleware 
As streamlog doesn't require it.
 6 months ago
Mike Lang be111ccb2a Change database primary key from UUID to TEXT 
We still store uuids, but in text form.
This allows us to store non-UUID ids for systems that have other ids.
 6 months ago
Mike Lang 72f7c59a77 Sheetsync: Split into the main loop logic + sheets-specific middleware 
NOTE ON CONFLICTS

In master, we moved sheets.py to common as it only contained a generic client.
Now sheets.py also contains specific sheetsync stuff.

Our resolution:
- Keep the generic version in common
- Keep the old version verbatim (including the now-redundant generic client) in sheetsync

We will move the sheetsync implementation to the generic client after the rebase is complete.
 6 months ago
Mike Lang 0e5bf1a0fe sheetsync: Split playlist runloop from normal sheets 6 months ago
Mike Lang a16259e892 sheetsync: Move id allocation out of sync_row() 6 months ago
Mike Lang 256e0f7ba1 sheetsync: Move row_index variable into row dict 6 months ago
Mike Lang c2d2f4b85c Revert "sheetsync: Support archive sheet" 
This reverts commit b93597c274.
 6 months ago
Mike Lang 4c87ad6735 Revert "sheetsync: unmapped columns aren't a problem." 
This reverts commit 5256577d00.
 6 months ago
ZeldaZach 8bbc72184c Support hot reload of Zulip Schedule 
- Move sheets API into common dir, since multi use
- Live download from Google Sheets using Config
- Falls back on old schedule if new one can't be downloaded for some reason
 7 months ago
Mike Lang 5256577d00 sheetsync: unmapped columns aren't a problem. 1 year ago
Mike Lang b93597c274 sheetsync: Support archive sheet 1 year ago
Mike Lang 1596feef1f sheetsync: Treat end time "--" as same as start time 
This is a common idiom, which we previously treated like a blank end time
(no end time set yet) but it makes more sense to treat as "same as start".
 2 years ago
Mike Lang 92ea0fbb77 sheetsync: even more hard-coded columns in database fetch 2 years ago
Mike Lang 29e6b9ead3 lists aren't sets 2 years ago
Mike Lang 546572a697 sheetsync: Don't pull the entire row from the database 
only the columns you need.

This matters because the thumbnail columns are very large and
we're transfering GB of data every time.
 2 years ago
Mike Lang db843c8f63 sheetsync: Report sync duration 2 years ago
Mike Lang 7dfb7b2544 sheetsync: Fix a bug where only show-in-description playlists were detected 
Because a blank 5th column would make sheetsync ignore the row.
 2 years ago
Mike Lang dd8385ccd8 sheetsync: Special case "<all>" in playlist tags to mean [] 
this avoids empty string meaning [] which is dangerous since it's easy to write accidentially.
 2 years ago
Mike Lang e7d1212085 fix typo 2 years ago