You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Mike Lang 1add3c5c22 Implement tombstoning to allow for segment deletion ... Rarely, we find ourselves needing to explicitly delete some data, eg. something that shouldn't have been public and should be removed from all records. It would also be nice if we could "clean up" bad versions of the same segment, which occasionally come up when downloaders have issues. With our distributed segment database, this is actually rather difficult as deleting the data from any one server would cause it to be restored from the others. It was only possible by stopping all backfill, deleting the data on all servers, then starting backfill again. Here we introduce a more practical approach. An operator creates an empty flag file with the same name as the segment to be deleted, but with a `.tombstone` extension. eg. to delete a file `/segments/desertbus/source/2019-11-13T02/45:51.608000-2.0-full-7IS92rssMzoSBQDIevHStbTNy-URRV3Vw-jzZ6pwOZM.ts`, you would create a tombstone `/segments/desertbus/source/2019-11-13T02/45:51.608000-2.0-full-7IS92rssMzoSBQDIevHStbTNy-URRV3Vw-jzZ6pwOZM.tombstone`. These tombstone files do two important things: * They hide the segment from being listed, which both means: * It can't be restreamed or put into a video * It can't be backfilled to other nodes * The tombstone files themselves do get backfilled to other nodes, so you only need to mark them on one server. Once the tombstone has propagated to all nodes, the segment file can be deleted independently on each one. We chose not to have a tombstone automatically trigger a segment deletion for safety reasons.		2 years ago
..
__init__.py	Improve WSGIServer graceful shutdown handling	3 years ago
cached_iterator.py	cache results of common.segments.best_segments_by_start	3 years ago
database.py	Set up plumbing for multi-range videos and implement no-transition fast cut videos only	3 years ago
dateutil.py	Audit and fix all usage of dateutil	6 years ago
flask_stats.py	Remove flask response size tracking	5 years ago
googleapis.py	common.googleapis: Fix a bug in retrying failed access token get	5 years ago
requests.py	more py3 fixes	3 years ago
segments.py	Implement tombstoning to allow for segment deletion	2 years ago
stats.py	py3 fixes for common	3 years ago