henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
437 Commits
34 Branches
0 Tags
19 MiB
Commit Graph

85 Commits (a36124914543b3f87e2699acd559ecf351074ff1)

Author SHA1 Message Date
Christopher Usher a361249145 renamed delete_before to keep_hours
Christopher Usher be562b8448 added warnings if keeping fewer hours than backfilling
Christopher Usher 9d81569d98 Added the ability to delete old hours
Mike Lang 7183b25ce9
Merge pull request from ekimekim/mike/database-resilience 
Changes to improve behaviour if the DB is down
Christopher Usher 929308f3e7 started on the segment_coverage service
Mike Lang 0e437566aa backfiller: Don't crash on DB errors 
We move all connection handling into get_nodes().
This means that problems connecting won't cause further errors
and cause the application to completely crash.

In turn, this means that the behaviour if the database goes down becomes
"continue backfilling from the nodes we know about" instead of crashing.
Christopher Usher ccb7f3c684 now use parse_segment_path to get hash from filename
Christopher Usher 36da1926d0 fixes for ekims suggestions
Christopher Usher 34785d1179 now checking the hashes
Christopher Usher b0562495d2 reject mismatched hashes; more metrics
Christopher Usher 120a5a7de0 started on checking the hash
Chris Usher 557ddddc31 better logging for the backfiller
Christopher Usher 44f0e0defb changed it back so only the name is checked
Chris Usher aab46f9765 fixed localhost bug in backfiller
Christopher Usher 84270c02ec logging fix
Christopher Usher fdb5d20db7 fix to database logging
Christopher Usher 497845f2da typos in comments
Christopher Usher 361e577474 fixes based on ekimekims suggestions
Christopher Usher 720684a388 refactoring to have consistent terminology
Christopher Usher 6d38250674 starting to refactor stream to channel and variant to quality
Mike Lang f50276bd01 backfiller: Expose recent_cutoff as CLI arg and increase it to 120s default 
In testing, GDQ's stream delay went up over 1min, which caused backfillers to backfill
segments at the same time they were downloaded. We increase the window for now,
and also make it configurable.
Mike Lang 29040a166c backfiller: Allow multiple concurrent segment downloads 
This will signifigantly increase throughput when downloading
large ranges of segments.

The max concurrency is exposed as a cli arg.

We also slightly modify the logged info, so it reports segments downloaded,
not just number of missing segments (which we might skip downloading for various reasons).
Christopher Usher 37bad7d5ed Also reset database connection on error in the backfiller
Mike Lang 7179fcacec Backfiller: ignore temp segments 
To make this work, we make type a proper segment field.

We also tell get_best_segments to ignore temp segments, since they might go away
before we can actually use them.
Christopher Usher dd246e1343 ekimekim's suggestions
Christopher Usher 9b28765ff2 Bug fixes to get the database connection working
Christopher Usher 4b9fbcb7d2 backfiller database code
Mike Lang f8d10dacdf Audit and fix all usage of dateutil 
We wrap direct dateutil calls to handle two distinct cases:

* `common.dateutil.parse()`: We want to handle arbitrary timestamps including tz info,
then convert them to UTC.

This is used in HLS parsing, and for command line input for backfiller

* `common.dateutil.parse_utc_only()`: We want to only handle UTC timestamps,
but datetime.strptime isn't flexible enough (eg. can't handle missing fractional component).

This is used for restreamer request params.
Christopher Usher 072e51f287 Renaming a variable that should have been part of the last commit
Christopher Usher 61107346c8 Fixed backing off on exceptions and some more documenation
Christopher Usher 728adb7c1d improvements suggested by ekim
Christopher Usher 530b9f7d5e more improvements based on ekims comments
Chris Usher 332e03de80 started in on ekim's comments
Christopher Usher 2857d3fb9f comments and some whitespace handling
Christopher Usher 4e6dbe1c74 Added localhost option to backfill to avoid backfilling the local machine
Christopher Usher ade0ad3d18 rewrite of get_nodes to allow getting list of files from a file
Christopher Usher 23fea7b154 bug fixing after testing
Christopher Usher 149974ce54 added multiple streams by largely copy and pasting the code from the 
downloader
Christopher Usher e4364b75b1 options to change where the node list is coming from
Christopher Usher baae0f1ac1 bug fix in arg list
Christopher Usher 65143a8ca2 more flexability for start time
Christopher Usher a8cb1ff370 fixed start not propagating to list_hours plus some refactorting
Christopher Usher 57bb74632f I should test these changes soon
Christopher Usher 64bc76c48b error handling I guess
Christopher Usher 09368d92e1 fixes and improvements suggested by ekimekim 
* simplied the backfiller local - now just a full backfill every couple minuteso
Christopher Usher 4eac6189ce backfiller working in parallel
Christopher Usher f4385ad4e3 hopefully did break anything with this refactor
Christopher Usher 1f53fa8d29 Bug fixes and logging improvements to the backfiller
Christopher Usher c9f6ee95c5 clean up for new gevent based backfiller.
Christopher Usher 7d9a5b4626 added workers and a worker manager