You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
wubloader/downloader/downloader
Mike Lang 21d5548980 Add new segment type "suspect"
We've noticed that when nodes have connection problems, they get full segments
with different hashes. Inspection of these segments shows that
they all have identical data up to a point.

Segments that fetched normally will then have the remainder of the data.
Segments that had issues will have a slightly corrupted end.
The data is still valid, and no errors are raised. It just doesn't have all the data.

We noticed that these corrupted segments all were cut off exactly 60sec after their requests
began. We believe this is a server-side timeout on the request that returns whatever data
it has, then closes the container file cleanly before returning successfully.

We detect segments that take > 59 seconds to recieve, and label them as "suspect".

Suspect segments are treated identically to partial segments, except they are always preferred
over partials.
5 years ago
..
__init__.py Add skeleton of downloader service 6 years ago
__main__.py Allow setting of log level via WUBLOADER_LOG_LEVEL env var 6 years ago
hls_playlist.py Ignore segments that are marked as ads 6 years ago
main.py Add new segment type "suspect" 5 years ago
twitch.py Downloader: Change access_token call to match website 5 years ago