76 Commits (eebfa5885b77194323ce2b9452a195bc04fc1362)

Author	SHA1	Message	Date
Mike Lang	b85296a81e	sheetsync: Move column indexes to match updated sheet ... New tags column shunts all columns after it right by 1. We will later want to parse that, but for now we ignore it.	4 years ago
Mike Lang	59d0fa3e40	sheetsync: Don't mis-parse blank as bad time	5 years ago
Mike Lang	ab157afe20	sheetsync: Clear event counts before each update ... Otherwise, no count of 0 ever gets set, and things are left showing values when they shouldn't.	5 years ago
Mike Lang	89a9e5554c	sheetsync: Record counts of rows in the DB, segmented by various columns ... This lets us view a number of useful graphs in dashboards, eg. rows by state, errored rows, rows by day, rows by category, meltdowns per day, fraction of events that are poster moments by category. Sheetsync was the natural place to do this since it was already periodically scanning the entire events table.	5 years ago
Mike Lang	1c0f3a627b	sheetsync: Log what worksheets got synced ... it's kinda important	5 years ago
Mike Lang	8b25f8be95	sheetsync: Inject an error into the error column if we fail to parse an input column	5 years ago
Mike Lang	8dc7b80de9	sheetsync: Improve timing of main loop ... Instead of always waiting 5 seconds between runs, wait until 5 seconds after the previous run started. This ensures we actually run every 5sec and not every 5sec + how long it takes to run	5 years ago
Mike Lang	cda8078f64	sheetsync: Only check the most recently changed two sheets most times ... Only check the other sheets every 4th time (20sec instead of 5sec). This elminiates a huge source of unnessecary reads, which prevents us from going over our API limit.	5 years ago
Mike Lang	4f6f4cad8b	sheetsync: Fix typos with metrics	5 years ago
Mike Lang	731ef9e2d0	Refactor dockerfiles for more shared layers ... By carefully ensuring most of our dockerfiles are identical in their first few layers, we only need to build those layers once instead of every time. In particular, we move installing gevent to before installing common, so that even when common changes gevent doesn't need to be reinstalled. This is important because gevent takes ages to install. Also fixes segment_coverage, which wasn't being installed.	5 years ago
Mike Lang	c740090c53	sheetsync: Add more metrics	5 years ago
Mike Lang	52e6c4ad41	sheetsync, cutter: Collect metrics on http calls ... In particular, to google apis.	5 years ago
Mike Lang	17af1c4e89	cutter, sheetsync: Wait for DB to connect on startup ... This is a nicer error than crashing in the depths of some error handler (which is what will happen if the DB goes unavailable while they're running), and it's a far more common case (eg. the DB is misconfigured) than having it fail halfway through. Neither of these services can do anything meaningful without the DB, so crashing without it is acceptable behaviour.	5 years ago
Mike Lang	48593e2b06	database, sheetsync: Add worksheet name column 'sheet_name' ... This tells us which sheet a row came from (so we don't need to scan every sheet to find it if we're trying to do lookups in that direction). It is also needed in order to tag the videos with the Day number.	5 years ago
Christopher Usher	f9ce41ef32	metrics for the sheetsync	5 years ago
Christopher Usher	a2e47b98f5	fixes -- in dates and the lack of the preshow	5 years ago
Christopher Usher	027c2900e2	fixes in response to ekim's comments	5 years ago
Christopher Usher	1dbe585837	retry database connection if it fails	5 years ago
Mike Lang	f7b591e78b	sheetsync: Log more information on HTTPError ... The api gives additional detail that we want to know when debugging.	5 years ago
Mike Lang	fe68e98804	sheetsync: Fix a failure mode where we never recover from a DB conn failure ... Since we never got a new conn after failure, we would just keep erroring with "connection already closed" errors.	6 years ago
Mike Lang	018e920808	sheet-sync: Some fixes	6 years ago
Mike Lang	f354130434	sheetsync: Only allocate ids when first needed ... This prevents rate limiting issues when immediately allocating all 999 ids for an empty sheet.	6 years ago
Mike Lang	11fc67f071	sheetsync: Review feedback ... * Expand on some comments * Fix conflicting port number * Write help text for all args	6 years ago
Mike Lang	9762f308a0	Implement main part of sheet sync	6 years ago
Mike Lang	5a44bfdf51	Google sheets api wrapper ... Exposes a way to read all rows, and write a single cell. We need to read all columns of each row so we know what would be modified so we only do updates to single cells that aren't already the correct value. This keeps us from impacting the sheet load too much with constantly changing values, which I think might be a thing even if the values are the same.	6 years ago
Mike Lang	2b4d2cce90	sheet sync: Basic skeleton	6 years ago