wubloader

Author	SHA1	Message	Date
Mike Lang	f0546e2ee3	Pin gevent to 1.5a2 to avoid https://github.com/gevent/gevent/issues/1711	3 years ago
Mike Lang	73d5941e05	downloader: Track timestamp of latest segment This gives us a "stream delay" metric. Prom doesn't have any native way to check the current value of a metric, in order to take max(). It only offers increment and set. We reach into some internals to do this in a hacky way, but the cleaner way would be to track the value ourselves and have a prom callback that gets the value. Sigh, I hate this prom library. I might write my own that's less dumb.	6 years ago
Mike Lang	6c3501db6f	downloader: Fix dateutil lib, which is actually called python-dateutil	6 years ago
Mike Lang	f193bd0f54	Re-write downloader to be resilient to failures as much as possible This makes the code crazy complicated and messy, but means we can be persistent about not giving up, while still retrying at the same time, and trying multiple urls at once until we find one that works. See docstrings for a full discussion on some of the failures we're trying to work around.	6 years ago
Mike Lang	439b623599	Add skeleton of downloader service	6 years ago