92 Commits (3cdd8f22ad01c68fd9cf18c4801b838be4329c4f)

Author	SHA1	Message	Date
Mike Lang	3cdd8f22ad	Split SheetSync object to be fully generic and have an EventsSync object for events	3 months ago
Mike Lang	6a9884ee5b	sheetsync: Move extra metrics columns into a variable	3 months ago
Mike Lang	37dd7c3e4f	sheetsync: Move various config fields to be defined on the class object ... Also make the columns lists into sets	3 months ago
Mike Lang	5559dffec6	sheetsync: pull metric gathering into a seperate function	3 months ago
Mike Lang	37f42853ec	sheetsync: Extract db table name into a variable ... So we can swap it out for playlists later	3 months ago
Mike Lang	00d0ecc3bb	sheetsync: Handle sheet_name as an input column to avoid special casing ... sheet_name must always be present on a sheet row, but is only present on a db row if the db actually stores that data. As a side benefit, the db will now update if sheet_name changes.	3 months ago
Mike Lang	2db20d202b	sheetsync: Update streamlog middleware for section -> tab rename	3 months ago
Mike Lang	9fb356bf45	sheetsync: Better error handling for running out of space when creating rows	3 months ago
Mike Lang	29ff11457e	sheetsync: Namespace all logs and metrics behind a sheetsync "name" ... This helps differentiate the multiple syncs we now have and will have: - syncing events from streamlog - reverse syncing events to sheets - syncing playlists	3 months ago
Mike Lang	87b504a00a	sheetsync: Rename "row" and "event" to "sheet_row" and "db_row" ... First step to combining event and playlist sync into one codepath. No actual behaviour changes intended.	3 months ago
Mike Lang	20ee79cdb1	Get sheets working again	3 months ago
Mike Lang	f89ab6fa43	Don't make sheet name an input column, go back to special casing it on row create ... This means it won't update if put in the wrong place, but avoids issues with reverse sync trying to write it out when it's not an actual column	3 months ago
Mike Lang	430938dc49	error is always a string, it just might be empty	3 months ago
Mike Lang	f8d3eb7f00	wip:	3 months ago
Mike Lang	ee4a68af50	clear up confusion with empty string vs None	3 months ago
Mike Lang	3e873ca5f6	wip: fixes	3 months ago
Mike Lang	eebfa5885b	sheetsync: pass in event id instead of event name	3 months ago
Mike Lang	cf41f572f5	Fix streamlog formatting	3 months ago
Mike Lang	986a1db964	sheetsync: Change how options are specified to allow multiple backends / syncs	3 months ago
Mike Lang	74869de89d	Implement reverse sync mode ... This is a mode where all data flows one-way from the database to the sheet. It is intended to be used to populate an empty sheet from database events, possibly sourced from somewhere else. To make this work, a few changes were required: * Track which ids we've seen so we know what events were not matched with a row * Allow `row` to be None in sync_rows * When it is, call the middleware to create a new row with a new id * In sheets, this is implemented by tracking the last empty rows we saw, and claiming them as needed.	3 months ago
Mike Lang	85de9757f7	sheetsync: Remove pick_worksheets() from middleware api ... Instead, get_rows() makes that decision internally if needed.	3 months ago
Mike Lang	17463d70fe	sheetsync: Remove worksheet from middleware apis ... since it's now baked into the row dict	3 months ago
Mike Lang	eec58f2651	sheetsync: Always have sheet name as part of row dict	3 months ago
Mike Lang	fa9a4b70bb	bugfix	3 months ago
Mike Lang	ca3f92c0b6	sheetsync: Use streamlog section instead of deriving day from start time	3 months ago
Mike Lang	071cd29f4d	sheetsync: Implement Streamlog middleware	3 months ago
Mike Lang	d064522d60	sheetsync: Move edit url management into Sheets middleware ... As streamlog doesn't require it.	3 months ago
Mike Lang	be111ccb2a	Change database primary key from UUID to TEXT ... We still store uuids, but in text form. This allows us to store non-UUID ids for systems that have other ids.	3 months ago
Mike Lang	72f7c59a77	Sheetsync: Split into the main loop logic + sheets-specific middleware ... NOTE ON CONFLICTS In master, we moved sheets.py to common as it only contained a generic client. Now sheets.py also contains specific sheetsync stuff. Our resolution: - Keep the generic version in common - Keep the old version verbatim (including the now-redundant generic client) in sheetsync We will move the sheetsync implementation to the generic client after the rebase is complete.	3 months ago
Mike Lang	0e5bf1a0fe	sheetsync: Split playlist runloop from normal sheets	3 months ago
Mike Lang	a16259e892	sheetsync: Move id allocation out of sync_row()	3 months ago
Mike Lang	256e0f7ba1	sheetsync: Move row_index variable into row dict	3 months ago
Mike Lang	c5c9075f9e	Basic streamlog api	3 months ago
Mike Lang	c2d2f4b85c	Revert "sheetsync: Support archive sheet" ... This reverts commit b93597c274.	3 months ago
Mike Lang	4c87ad6735	Revert "sheetsync: unmapped columns aren't a problem." ... This reverts commit 5256577d00.	3 months ago
ZeldaZach	8bbc72184c	Support hot reload of Zulip Schedule ... - Move sheets API into common dir, since multi use - Live download from Google Sheets using Config - Falls back on old schedule if new one can't be downloaded for some reason	4 months ago
Mike Lang	3606fadaa8	Pin gevent version to work around build issues ... Seeing the following error on latest versions of gevent: Traceback (most recent call last): File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/lib/python3.9/site-packages/zulip_bots/schedulebot.py", line 2, in <module> import gevent.monkey File "/usr/lib/python3.9/site-packages/gevent/__init__.py", line 72, in <module> from gevent._hub_local import get_hub File "/usr/lib/python3.9/site-packages/gevent/_hub_local.py", line 150, in <module> import_c_accel(globals(), 'gevent.__hub_local') File "/usr/lib/python3.9/site-packages/gevent/_util.py", line 148, in import_c_accel mod = importlib.import_module(cname) File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ModuleNotFoundError: No module named 'gevent._gevent_c_hub_local'	1 year ago
Mike Lang	78c053000e	Upgrade pip in order to make wheels work	1 year ago
Mike Lang	5256577d00	sheetsync: unmapped columns aren't a problem.	1 year ago
Mike Lang	b93597c274	sheetsync: Support archive sheet	1 year ago
Mike Lang	044dfb8084	Pin argh to avoid stupid breaking changes	1 year ago
Mike Lang	30d5ccc483	Fix all old references to github.com/ekimekim/wubloader	1 year ago
Mike Lang	1596feef1f	sheetsync: Treat end time "--" as same as start time ... This is a common idiom, which we previously treated like a blank end time (no end time set yet) but it makes more sense to treat as "same as start".	1 year ago
Mike Lang	92ea0fbb77	sheetsync: even more hard-coded columns in database fetch	2 years ago
Mike Lang	29e6b9ead3	lists aren't sets	2 years ago
Mike Lang	546572a697	sheetsync: Don't pull the entire row from the database ... only the columns you need. This matters because the thumbnail columns are very large and we're transfering GB of data every time.	2 years ago
Mike Lang	db843c8f63	sheetsync: Report sync duration	2 years ago
Mike Lang	7dfb7b2544	sheetsync: Fix a bug where only show-in-description playlists were detected ... Because a blank 5th column would make sheetsync ignore the row.	2 years ago
Mike Lang	dd8385ccd8	sheetsync: Special case "<all>" in playlist tags to mean [] ... this avoids empty string meaning [] which is dangerous since it's easy to write accidentially.	2 years ago
Mike Lang	e7d1212085	fix typo	2 years ago