mirror of https://github.com/ekimekim/wubloader
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
21d5548980
We've noticed that when nodes have connection problems, they get full segments with different hashes. Inspection of these segments shows that they all have identical data up to a point. Segments that fetched normally will then have the remainder of the data. Segments that had issues will have a slightly corrupted end. The data is still valid, and no errors are raised. It just doesn't have all the data. We noticed that these corrupted segments all were cut off exactly 60sec after their requests began. We believe this is a server-side timeout on the request that returns whatever data it has, then closes the container file cleanly before returning successfully. We detect segments that take > 59 seconds to recieve, and label them as "suspect". Suspect segments are treated identically to partial segments, except they are always preferred over partials. |
5 years ago | |
|---|---|---|
| .. | ||
| downloader | 5 years ago | |
| Dockerfile | 5 years ago | |
| setup.py | 5 years ago | |