Commit Graph

18 Commits (1cb4d413341a44c6a75f0031c5fddcd87eb0be60)

Author SHA1 Message Date
Mike Lang 4f6f4cad8b sheetsync: Fix typos with metrics 5 years ago
Mike Lang 731ef9e2d0 Refactor dockerfiles for more shared layers
By carefully ensuring most of our dockerfiles are identical in their first few layers,
we only need to build those layers once instead of every time.

In particular, we move installing gevent to before installing common,
so that even when common changes gevent doesn't need to be reinstalled.

This is important because gevent takes ages to install.

Also fixes segment_coverage, which wasn't being installed.
5 years ago
Mike Lang c740090c53 sheetsync: Add more metrics 5 years ago
Mike Lang 52e6c4ad41 sheetsync, cutter: Collect metrics on http calls
In particular, to google apis.
5 years ago
Mike Lang 17af1c4e89 cutter, sheetsync: Wait for DB to connect on startup
This is a nicer error than crashing in the depths of some error handler
(which is what will happen if the DB goes unavailable while they're running),
and it's a far more common case (eg. the DB is misconfigured) than having it fail
halfway through.

Neither of these services can do anything meaningful without the DB,
so crashing without it is acceptable behaviour.
5 years ago
Mike Lang 48593e2b06 database, sheetsync: Add worksheet name column 'sheet_name'
This tells us which sheet a row came from
(so we don't need to scan every sheet to find it if we're trying to do
lookups in that direction).

It is also needed in order to tag the videos with the Day number.
5 years ago
Christopher Usher f9ce41ef32 metrics for the sheetsync 5 years ago
Christopher Usher a2e47b98f5 fixes -- in dates and the lack of the preshow 5 years ago
Christopher Usher 027c2900e2 fixes in response to ekim's comments 5 years ago
Christopher Usher 1dbe585837 retry database connection if it fails 5 years ago
Mike Lang f7b591e78b sheetsync: Log more information on HTTPError
The api gives additional detail that we want to know when debugging.
5 years ago
Mike Lang fe68e98804 sheetsync: Fix a failure mode where we never recover from a DB conn failure
Since we never got a new conn after failure, we would just keep erroring with
"connection already closed" errors.
5 years ago
Mike Lang 018e920808 sheet-sync: Some fixes 5 years ago
Mike Lang f354130434 sheetsync: Only allocate ids when first needed
This prevents rate limiting issues when immediately allocating all 999 ids
for an empty sheet.
5 years ago
Mike Lang 11fc67f071 sheetsync: Review feedback
* Expand on some comments
* Fix conflicting port number
* Write help text for all args
5 years ago
Mike Lang 9762f308a0 Implement main part of sheet sync 5 years ago
Mike Lang 5a44bfdf51 Google sheets api wrapper
Exposes a way to read all rows, and write a single cell.

We need to read all columns of each row so we know what would be modified
so we only do updates to single cells that aren't already the correct value.

This keeps us from impacting the sheet load too much with constantly changing values,
which I think might be a thing even if the values are the same.
5 years ago
Mike Lang 2b4d2cce90 sheet sync: Basic skeleton 5 years ago