mirror of https://github.com/yt-dlp/yt-dlp
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
a9ac178eb1
I use yt-dlp on Windows writing to a Linux system via SMB over a 10GbE connection and downloading via 400 Mbps cable internet. I have observed that downloads often seem to start very fast (40+ MiB/sec) but then throttle down to 8-20 MiB/sec. I also observed a large amount of disk thrashing for such a large array and small amount of data that's supposedly being written sequentially. The problem is two-fold. Downloaded fragments are stored using a very short-lived *-FragX file, then immediately appended to the stream upon fragment completion, and deleted. Both operations use small write buffers. When the OS write buffers start to flush, the two sets of writes plus the large number of writes start to force competition to complete the queued writes in different areas of the volume. Python defaults to sending writes at the underlying device's "block size" or a fallback to io.DEFAULT_BUFFER_SIZE. In practical terms, this means a write buffer of 4096 or 8192 bytes. This commit increases most write buffers to 65536 (64 KiB) using the open() buffering=X option, significantly speeding up writes of larger chunks of data and reducing potential fragmentation in low disk space conditions. With these changes, I consistently see fast downloads and the array thrashing is noticeably lessened. |
6 days ago | |
|---|---|---|
| .. | ||
| __pyinstaller | ||
| compat | 1 month ago | |
| dependencies | 2 months ago | |
| downloader | 3 weeks ago | |
| extractor | 6 days ago | |
| networking | 3 months ago | |
| postprocessor | 3 weeks ago | |
| utils | 6 days ago | |
| YoutubeDL.py | 6 days ago | |
| __init__.py | 3 weeks ago | |
| __main__.py | ||
| aes.py | 3 weeks ago | |
| cache.py | 7 months ago | |
| cookies.py | 3 weeks ago | |
| jsinterp.py | 5 months ago | |
| minicurses.py | ||
| options.py | 1 month ago | |
| plugins.py | 3 weeks ago | |
| socks.py | 7 months ago | |
| update.py | 1 week ago | |
| version.py | 2 weeks ago | |
| webvtt.py | 7 months ago | |