72 Commits (condor-scripts)

Author	SHA1	Message	Date
Christopher Usher	a63e18c0a9	added state to upload_errors	5 years ago
Christopher Usher	228468749c	changes based on ekim's suggestions	5 years ago
Christopher Usher	5c08540c0a	stats for the cutter	5 years ago
Christopher Usher	989ef80d30	starting on adding some stats to the cutter	5 years ago
Christopher Usher	fec8cff185	Merge branch 'master' into chrusher-database-replication	5 years ago
Christopher Usher	027c2900e2	fixes in response to ekim's comments	5 years ago
Christopher Usher	23e3cfce20	Added editor, edit_time and upload_time to thrimshim and cutter updates of the ... database	5 years ago
Christopher Usher	1dbe585837	retry database connection if it fails	5 years ago
Mike Lang	e4d3e418c8	transcode checker: longer retry while waiting for videos to finish ... but still check db often. This prevents us from using too much api quota on these checks, while still letting us spot new videos quickly.	5 years ago
Mike Lang	1f15900b6f	cutter: At least for now, don't auto-retry errors ... This leads to rapidly exhausting our upload limit since even a fast failed request costs the same amount of usage quota as a 1-hour long video.	5 years ago
Mike Lang	fbef4725d7	cutter: Handle case where we are told to stop while looking for candidates ... Previously, it would return None and things would break. Now the None is handled correctly, and is documented.	5 years ago
Mike Lang	5cec6ec96e	cutter: Reconnect after any error that might be a database error ... After certain kinds of DB error (eg. lost conn), we need to make a new conn to have things work again. To be safe, we just do it after every error where it might be a problem.	5 years ago
Mike Lang	fea9ff6c1d	cutter: Fix dockerfile, which was missing ffmpeg dependency	5 years ago
Mike Lang	e048db0d94	cutter: Fix a failure mode where we never recover from a DB conn failure in TranscodeChecker ... Since we never got a new conn after failure, we would just keep erroring with "connection already closed" errors. This isn't applicable to the main cutter loops since a DB failure there will restart the process.	5 years ago
Mike Lang	85c110ccb4	cutter: Fix typo from when we moved to the client model instead of auth headers	5 years ago
Mike Lang	3647d091f8	Move common google api auth functionality into common ... So we can reuse it for google sheets	5 years ago
Mike Lang	5b2a1ef6b7	cutter: Implement actual cut methods ... Each method is fairly complicated, but is self-contained and can be examined independently. cut_jobs in particular contains several extra helpers and directs control flow via some iterators. This is unfortunately nessecary due to the requests interface.	6 years ago
Mike Lang	ae809c696c	cutter: Outline of how main cutter run loop works ... This commit only lays out the main loop, showing the high-level flow and defining shared utilities. This is for clarity. The actual methods that do the work will be implemented seperately.	6 years ago
Mike Lang	80c1a66aa0	cutter: Implement TranscodeChecker ... It runs on an interval, fetching all videos in TRANSCODING from the DB, checking them against youtube, and then updating any that are done. It should be noted that youtube somewhat lies about what being "done" means, but this is a better approximation than nothing.	6 years ago
Mike Lang	3ce8360a1e	cutter: Add database manager and connections ... One connection each for transcode checker and cutter. We don't need more than one each since both workers only ever do one thing at once.	6 years ago
Mike Lang	fdd245a6d9	cutter: Add lightweight youtube client ... Provides basic youtube api calls, and gets passed into both transcode checker and cutter. The official youtube client library is many orders of magnitude larger and more complicated, and can't actually do what we want (stream an upload of unknown size).	6 years ago
Mike Lang	e4b6110fd7	cutter: Add initial outline ... The cutter has two jobs: * To cut videos, taking them through states EDITED -> TRANSCODING * To monitor TRANSCODING videos for when they're complete We run these as separate greenlets with their own DB connections, and if either dies we gracefully shut down the other.	6 years ago