Commit Graph

78 Commits (8264206f0999a19b131ce816b1e5b783d119fc87)

Author SHA1 Message Date
Mike Lang 47ff92b155 downloader: Fix bug where mark_working wasn't called
This meant that old workers would never shut down, causing us to fetch the same media playlist
and same segments multiple times for no reason, and to never give up in face of (non-403/404) errors
even once we have something else working.
6 years ago
Mike Lang 3042d00516 downloader: Give up on 404 in addition to 403
Also fix some logging.

When we're out of touch with twitch for long enough, our segment URL will get
so old that twitch stops returning 403 because our token is expired,
and start returning 404s, presumebly becasue the underlying resource has gone away.

We want to treat these the same.
6 years ago
Mike Lang 7f9a1dbe45 downloader: Remove implicit source quality arg
This brings it in line with backfiller, is more flexible and less surprising
6 years ago
Mike Lang 0d627715f3 downloader: Track number of downloaded segments
This is the most important metric, we can add more later.
6 years ago
Mike Lang b4b315b6bc Expose prometheus metrics for backfiller and downloader 6 years ago
Mike Lang b0ded641c3 Add a logging handler which counts logs for prometheus stats
This isn't as good as having a full centralised logging system, but should
suffice to know if anything funny is happening.
6 years ago
Mike Lang 17972b87aa Allow setting of log level via WUBLOADER_LOG_LEVEL env var
By using an env var, it is universal and happens prior to arg parsing,
at the same point we do other logging setup.
6 years ago
Mike Lang c0357680cf downloader: Use caller's logger inside soft_hard_timeout 6 years ago
Mike Lang a628676e74 downloader: Log to subloggers instead of the root logger
This gives us some context when logging, and is best practice.
6 years ago
Mike Lang 6815924097 Fix some bugs and linter errors introduced by backfiller
I ran `pyflakes` on the repo and found these bugs:

```
./common/common.py:289: undefined name 'random'
./downloader/downloader/main.py:7: 'random' imported but unused
./backfiller/backfiller/main.py:150: undefined name 'variant'
./backfiller/backfiller/main.py:158: undefined name 'timedelta'
./backfiller/backfiller/main.py:171: undefined name 'sort'
./backfiller/backfiller/main.py:173: undefined name 'sort'
```
(ok, the "imported but unused" one isn't a bug, but the rest are)

This fixes those, as well as a further issue I saw with sorting of hours.

Iterables are not sortable. As an obvious example, what if your iterable was infinite?
As a result, any attempt to sort an iterable that is not already a friendly type like a list
or tuple will result in an error. We avoid this by coercing to list, fully realising the iterable
and putting it into a form that python will let us sort. It also avoids the nasty side-effect
of mutating the list that gets passed into us, which the caller may not expect. Consider this example:

```
>>> my_hours = ["one", "two", "three"]
>>> print my_hours
["one", "two", "three"]
>>> backfill_node(base_dir, node, stream, variants, hours=my_hours, order='forward')
>>> print my_hours
["one", "three", "two"]
```

Also, one of the linter errors was non-trivial to fix - we were trying to get a list of hours
(which is an api call for a particular variant), but at a time when we weren't dealing with a single
variant. My solution was to get a list of hours for ALL variants, and take the union.
6 years ago
Christopher Usher fec0975d18 fixed white space and the like 6 years ago
Christopher Usher 3cdfaad664 moved rename, ensure_directory and jitter to common
Move a few useful functions in downloader used in the backfiller to common
6 years ago
Mike Lang 1dce14bf77 downloader: Fix and improve the stop mechanism, stop on SIGTERM
Allows for graceful shutdown
6 years ago
Mike Lang 7b10429846 downloader: Dockerfile fixes to make it work 6 years ago
Mike Lang 6c3501db6f downloader: Fix dateutil lib, which is actually called python-dateutil 6 years ago
Mike Lang 7257fb9b73 downloader: Include channel name in path, instead of assuming it's already in base_dir
Previously, downloader would put files under BASE_DIR/VARIANT/HOUR/FILE.ts
now, it will put files under BASE_DIR/STREAM/VARIANT/HOUR/FILE.ts

This brings downloader in line with restreamer's concept of base_dir
6 years ago
Christopher Usher efe30c1942 Added a comment to highlight recursion 6 years ago
Mike Lang 75c9793eac Remove central config file as it's more trouble than it's worth
Simpler and easier for testing to stick to configuration via CLI args.
We'll worry about deployment later.
6 years ago
Mike Lang 031dd60897 downloader: Fix some typos around the max age calculation 6 years ago
Mike Lang 6377db2aa2 downloader: Bug fixes and improvements
* Fix bug where soft timeout is not cancelled if an exception occurs
* Various logging tweaks
* Prevent master playlist wait time from going negative
* Stop gracefully if stream worker detects end of stream
* Don't treat master playlist 404 as an error, it just means the stream isn't up
6 years ago
Mike Lang 6e0dcd5e22 downloader: Fix bugs and missing bits in initial implementation
* Set a reasonable log format
* Make soft timeouts not always fire
* Change soft_hard_timeout signature slightly for ease-of-use
* Make renames not fail if file already exists
* Misc typos
6 years ago
Mike Lang f193bd0f54 Re-write downloader to be resilient to failures as much as possible
This makes the code crazy complicated and messy, but means we can be persistent about
not giving up, while still retrying at the same time, and trying multiple urls at once
until we find one that works.

See docstrings for a full discussion on some of the failures we're trying to work around.
6 years ago
Mike Lang 10241b6190 Downloader: Implement a very basic proof of concept version
Missing a LOT of tankiness, ways to configure, conversion to bustime, etc.
6 years ago
Mike Lang 8993773a22 downloader.twitch: Deals with twitch specifics of playlist management 6 years ago
Mike Lang 961712b919 downloader: Import hls_playlist from streamlink
This is a useful library and we might as well use it.
Copying it over and slightly modifying it to work was easier than importing all of streamlink.

The original version may be found at 30043408c7/src/streamlink/stream/hls_playlist.py
6 years ago
Mike Lang 30612f00ad downloader: basic startup path 6 years ago
Mike Lang 1b21694c27 Add a simple build script to build docker images and a basic dockerfile 6 years ago
Mike Lang 439b623599 Add skeleton of downloader service 6 years ago