Commit Graph

205 Commits (9bcfe33be7f1aa7164e690ced133cae4b063efa4)

Author SHA1 Message Date
pukkandan 49a57e70a9
[cleanup] misc
pukkandan d3c93ec2b7
Don't create console for subprocesses on Windows ()
Closes 
pukkandan b4b855ebc7
[fragment] Print error message when skipping fragment
pukkandan 2cda6b401d
Revert "[fragments] Pad fragments before decrypting ()"
This reverts commit 373475f035.
pukkandan aa7785f860
[utils] Standardize timestamp formatting code
Closes 
pukkandan 9fab498fbf
[http] Retry on socket timeout
Closes 
shirt 373475f035
[fragments] Pad fragments before decrypting ()
Closes , , 
Authored by: shirt-dev
pukkandan 580d3274e5 [youtube] Expose different formats with same itag
pukkandan 03b4de722a
[downloader] Fix slow progress hooks
Closes 
pukkandan d5a39f0bad
[http] Show the last encountered error
Closes 
pukkandan fc5c8b6492
[eria2c] Fix --skip-unavailable fragment
pukkandan c111cefa5d
[downloader/ffmpeg] Improve simultaneous download and merge
pukkandan 21186af70a
[downloader] Fix throttledratelimit
The timer should not reset at start of each block
pukkandan d1d5c08f29
[minicurses] Fix when printing to file
Closes 
pukkandan b5ae35ee6d
[cleanup] Misc cleanup
pukkandan b922db9fe5
[http] Respect user-provided chunk size over extractor's
pukkandan 819e05319b
Improved progress reporting (See desc) ()
* Separate `--console-title` and `--no-progress`
* Add option `--progress` to show progress-bar even in quiet mode
* Fix and refactor `minicurses`
* Use `minicurses` for all progress reporting
* Standardize use of terminal sequences and enable color support for windows 10
* Add option `--progress-template` to customize progress-bar and console-title
* Add postprocessor hooks and progress reporting

Closes: , , , 
pukkandan 49e7e9c3ce
[docs,build] Change all pycryptodome references to pycryptodomex
shirt 7687c8ac6e
[HLS] Fix decryption issues ()
* Unpad HLS fragments with PKCS#7 according to datatracker.ietf.org/doc/html/rfc8216
* media_sequence should only be incremented in for media fragments
* The native decryption should only be used if ffmpeg is unavailable since it is significantly slower. Closes 

Authored by: shirt-dev, pukkandan
coletdjnz 99e9e001de
[youtube] Cleanup authentication code ()
Authored by: coletdjnz
pukkandan be867b03f5
bugfix for bd50a52b0d
The Hatsune Daishi ee2b3563f3
[downloader/niconico] Pass custom headers ()
Closes 
Authored by: nao20010128nao
The Hatsune Daishi bd50a52b0d
Basic framework for simultaneous download of multiple formats ()
Authored by: nao20010128nao
pukkandan d806c9fd97
[docs,cleanup] Add deprecation warning in docs
for some counter intuitive behaviour that may be removed in future.

and fix linter
pukkandan 1009f67c2a
[fragment,aria2c] Generalize and refactor some code
pukkandan d9d8b85747
[fragment] Fix range header when using `-N` and media sequence ()
Authored by: shirt
pukkandan daf7ac2b92
[fragment] Avoid repeated request for AES key
pukkandan 96933fc1b6
[aria2c] Fix IV for some AES-128 streams
Authored by: shirt
pukkandan 57aa7b8511
[hls] Byterange + AES128 is supported by native downloader
pukkandan edf65256aa
[hls,aes] Fallback to native implementation for AES-CBC
and detect `Cryptodome` in addition to `Crypto`

Closes 
Related: 
pukkandan d5fe04f5c7
Fix `--compat-option no-direct-merge`
Paul Wrubel d75201a873
Use `os.replace` where applicable ()
When using 
```py
os.remove(encodeFilename(filename))
os.rename(encodeFilename(temp_filename), encodeFilename(filename))
```
the `os.remove` need not be atomic and so can be executed arbitrarily compared to the immediately following rename call. It is better to use `os.replace` instead

Authored by: paulwrubel
pukkandan 691d5823d6
[aria2c] Obey `--rate-limit`
pukkandan 45842107b9
fix bug in 6251555f1c
:ci skip
pukkandan 6251555f1c
[downloader/ffmpeg] Support for DASH manifests (experimental)
Closes 
pukkandan 330690a214
[downloader/ffmpeg] Allow passing custom arguments before -i
Closes 
Felix S 7a6742b5f9
[webvtt] Fix timestamp overflow adjustment ()
In some streams, empty segments may appear with a bogus, non-monotone MPEG timestamp.
This should not be considered as an overflow

Authored by: fstirlitz
pukkandan f775c83110
Fix `--force-overwrites` when using `-k`
For formats that need merge, the `.fxxx` files are not removed before
downloading the corresponding `.part` files. This causes the rename to fail
Felix S 0715f7e19b
Revert erroneous use of the `Content-Length` header ()
This reverts commit 6c907eb33f

The use of the Content-Length value here is erroneous and may lead
to truncated downloads if a compression scheme is specified in the
Content-Encoding header, as the Content-Length header refers to the
size of encoded data, not of the raw bytestream. This has been noticed
in the wild with WebVTT subtitle segments.

Authored by: fstirlitz
Felix S 25a3f4f5d6
[webvtt] Merge daisy-chained duplicate cues ()
Fixes: https://github.com/yt-dlp/yt-dlp/issues/631#issuecomment-893338552

Previous deduplication algorithm only removed duplicate cues with
identical text, styles and timestamps.  This change also merges
cues that come in ‘daisy chains’, where sequences of cues with
identical text and styles appear in which the ending timestamp of
one equals the starting timestamp of the next.

This deduplication algorithm has the somewhat unfortunate side effect
that NOTE blocks between cues, if found, will be emitted in a different
order relative to their original cues.  This may be unwanted if perfect
fidelity is desired, but then so is daisy-chain deduplication itself.
NOTE blocks ought to be ignored by WebVTT players in any case.

Authored by: fstirlitz
pukkandan ad3dc496bb Misc fixes - See desc
* Remove unnecessary uses of _list_from_options_callback
* Fix download tests - Bug from 6e84b21559
* Rename ExecAfterDownloadPP to ExecPP and refactor its tests
* Ensure _write_ytdl_file closes file handle on error - Potential fix for 
pukkandan 3448870205
[docs] Fix some mistakes and improve doc
pukkandan b868936cd6 [cleanup] Misc
pukkandan 9cc1a3130a
Fix resuming when using `--no-part`
Closes 
nikhil bb36a55c41
[nbcolympics:stream] Fix extractor
PR: https://github.com/ytdl-org/youtube-dl/pull/29688
Closes: , https://github.com/ytdl-org/youtube-dl/issues/29665

* Livestreams are untested
* If using ffmpeg as downloader, v4.3+ is needed since `-http_seekable` option is necessary
* Instead of making a seperate key for each arg that needs to be passed to ffmpeg, I made `_ffmpeg_args`
* This deprecates `_seekable`, but the option is kept for compatibility

Authored by: nchilada, pukkandan
pukkandan e5611e8eda
[ffmpeg] Fix streaming `mp4` to `stdout`
pukkandan a46a815b05
[cleanup] Fix linter in 96fccc101f
pukkandan 96fccc101f [downloader] Allow streaming unmerged formats to stdout using ffmpeg
For this to work:
1. The downloader must be ffmpeg
2. The selected formats must have the same protocol
3. The formats must be downloadable by ffmpeg to stdout

Partial solution for: https://github.com/ytdl-org/youtube-dl/issues/28146, https://github.com/ytdl-org/youtube-dl/issues/27265
pukkandan dbf5416a20 [cleanup] Refactor some code
pukkandan f45e6c1126
[downloader] Pass same status object to all `progress_hooks`
pukkandan 3ba7740dd8
[downloader] Pass `info_dict` to `progress_hook`s
pukkandan 11f9be0912
[youtube] Extract data from multiple clients ()
* `player_client` accepts multiple clients
* default `player_client` = `android,web`
* music clients can be specifically requested
* Add IOS `player_client`
* Hide live dash since they can't be downloaded

Closes 

Authored-by: pukkandan, colethedj
pukkandan 723d44b92b
[fragment] Handle errors in threads correctly
pukkandan 6b6c16ca6c [downloader/ffmpeg] Fix `--ppa` when using simultaneous download
siikamiika c2603313b1
[youtube_live_chat] use `clickTrackingParams` ()
Authored by: siikamiika
pukkandan bd4d1ea398
[cleanup] Minor refactoring of `fragment`
pukkandan 8e897ed283
[fragment] Return status of download correctly
siikamiika d534c4520b
[youtube_live_chat] Fix download with cookies ()
Closes  

Authored by: siikamiika
siikamiika c60ee3a218
[youtube_live_chat] Support ongoing live chat ()
Authored by: siikamiika
pukkandan 8a77e5e6bc
[cleanup] Revert unnecessary changes in 51d9739f80
pukkandan 51d9739f80
Add option `--throttled-rate` below which video data is re-extracted
Currently only for HTTP downloads

Closes , workaround for https://github.com/ytdl-org/youtube-dl/issues/29326
pukkandan 4c7853de14
[fragment] Merge during download for `-N`, and refactor `hls`/`dash` ()
pukkandan e36d50c5dd
[websockets] Add `WebSocketFragmentFD` ()
Necessary for 

Co-authored by: nao20010128nao, pukkandan
pukkandan ff0f78e1fe
[aria2c] Lower `--min-split-size` for HTTP downloads
This makes downloading smaller files much faster
felix cdb19aa4c2
[downloader/mhtml] Add new downloader ()
This downloader is intended to be used for streams that consist of a
timed sequence of stand-alone images, such as slideshows or thumbnail
streams

This can be used for implementing:

https://github.com/ytdl-org/youtube-dl/issues/4974#issue-58006762
https://github.com/ytdl-org/youtube-dl/issues/4540#issuecomment-69574231
https://github.com/ytdl-org/youtube-dl/pull/11185#issuecomment-335554239

https://github.com/ytdl-org/youtube-dl/issues/9868
https://github.com/ytdl-org/youtube-dl/pull/14951


Authored by: fstirlitz
felix 2ec1759f9d
[downloader/ffmpeg] Hide FFmpeg banner unless in verbose mode ()
Authored by: fstirlitz
pukkandan e06ca6ddac
[hls] Decrypt fragment when reading from disk
Closes 
pukkandan d89da64b1d
[hls,dash] When using `concurrent_fragment_downloads`, do not keep the fragment content in memory
Partial fix for 
This is a temporary solution until  can be implemented
pukkandan 5dcd8e1d88
[hls] Disable external downloader for `webtt`
pukkandan 18e674b4f6
[ffmpeg] Download and merge in a single step if possible
pukkandan 856bb8f99d
[downloader] Fix `write_debug`
pukkandan 0760b0a7e2
Standardize `write_debug`
pukkandan 000ee7ef34
[fragment] Make sure first segment is not skipped
pukkandan e8e738406a
Add experimental option `--check-formats` to test the URLs before format selection
pukkandan 848887eb7a
[downloader] Fix `quiet` and `to_stderr`
pukkandan 6ef6bcbd6b
[fragment] Ensure the file is closed on error
The Hatsune Daishi 0fa9a1e236
[whowatch] Add extractor
closes 

Authored by: nao20010128nao 
Modified from: 9e4a0e061a/youtube_dl/extractor/whowatch.py
pukkandan eb55bad5a0
[aria2c] Fix whitespace being stripped off
Closes 
pukkandan 2291dbce2a
[niconico] Fix HLS formats
Closes 

* The structure of the API JSON was changed
* Smile Video seems to be no longer available. So remove the warning
* Move ping to downloader
* Change heartbeat interval to 40sec
* Remove unnecessary API headers

Authored-by: CXwudi, tsukumijima, nao20010128nao, pukkandan
Tested by: tsukumijima
Felix S 66a1b8643a [downloader/ism] Support muxing TTML subtitles
Felix S 15828bcf25 [downloader/hls] Handle MPEG-2 PES timestamp overflow
Felix S 333217f43e [downloader/hls] Remove duplicate cues using a sliding window of candidates
Felix S 4a2f19abbd [downloader/hls] Assemble single-file WebVTT subtitles from HLS segments
Felix S fd76a14259 [extractor/common, downloader/ism] Extract SSTR subtitle tracks
_parse_ism_formats was extended into _parse_ism_formats_and_subtitles;
all direct users were updated, though _extract_ism_formats was left
as a compatibility wrapper.

The SSTR downloader was also modified in order to prepare for muxing
subtitle streams, although no support for any subtitle codecs was
added in this commit.
Felix S d4553567d2 [downloader/ism] Prevent writing the header again when resuming an interrupted download
Felix S 4d49884c58 [downloader/fragment] Allow persisting extra state when a download is interrupted
pukkandan 88728713c8
Py2 compatibility for `FileNotFoundError`
pukkandan 33245766ab
[downloader] Fix ffmpeg selection for m3u8_native
pukkandan 72e1fe969f
[downloader] Fix downloader selection for m3u8
Bug introduced by: 52a8a1e1b9 and a31953b0e6
pukkandan a31953b0e6
[downloader] Fix external downloader selection for m3u8
Closes 
pukkandan 68379de561
[downloaders] Fix API access
Bug introduced by: 52a8a1e1b9
pukkandan 52a8a1e1b9
Option to choose different downloader for different protocols
* Renamed `--external-downloader-args` to `--downloader-args`
* Added `native` as an option for the downloader
* Use similar syntax to `--downloader-args` etc. Eg: `--downloader dash:native --downloader aria2c`
* Deprecated `--hls-prefer-native` and `--hls-prefer-ffmpeg` since the same can now be done with `--downloader "m3u8:native"` and `m3u8:ffmpeg` respectively
* Split `frag_urls` protocol into `m3u8_frag_urls` and `dash_frag_urls`
* Standardize shortening of protocol names with `downloader.shorten_protocol_name`
pukkandan 83b20a970d
[youtube_live_chat] fix bug when trying to set cookies
Closes 
pukkandan beb4b92a66
More consistent warning messages ()
Co-authored by: Damiano Amatruda <damiano.amatruda@outlook.com>
pukkandan fe845284c4
[aria2c] Support retry/abort unavailable fragments ()
Co-authored by: Damiano Amatruda <damiano.amatruda@outlook.com>
pukkandan 2b3bf01c90 [aria2c] Better default arguments
pukkandan e92caff5d5
Refactor (See desc)
* Create `FFmpegPostProcessor.real_run_ffmpeg` that can accept multiple input/output files along with switches for each
* Rewrite `cli_configuration_args` and related functions
* Create `YoutubeDL._ensure_dir_exists` - this was previously defined in multiple places
shirt 4cf1e5d2f9
Native concurrent downloading of fragments ()
* Option `--concurrent-fragments` (`-N`) to set the number of threads

Related: 

Known issues:
* When receiving Ctrl+C, the process will exit only after finishing the currently downloading fragments
* The download progress shows the speed of only one thread

Authored by shirt-dev
pukkandan 0a473f2f0f
More improvements to HLS/DASH external downloader code
* Fix error when there is no `protocol` in `info_dict`
* Move HLS byte range detection to `Aria2cFD` so that the download will fall back to the native downloader instead of ffmpeg
* Fix bug with getting no fragments in DASH
* Convert `check_results` in `can_download` to a generator
shirt-dev d7009caa03
Improve HLS/DASH external downloader code ()
Authored by: shirt
pukkandan 9e631877f8
[downloader] Fix bug for ffmpeg/httpie
Caused by: 7f7de7f94d
pukkandan 5ef7d9bdd8
Release 2021.03.01
pukkandan 7f7de7f94d Allow specifying path in `--external-downloader`
pukkandan 9bd2020476 [hls] Enable `--hls-use-mpegts` by default when downloading live-streams
* Also added option `--no-hls-use-mpegts` to disable this

Related: 
Pccode66 7a5c1cfe93
Completely change project name to yt-dlp ()
* All modules and binary names are changed
* All documentation references changed
* yt-dlp no longer loads youtube-dlc config files
* All URLs changed to point to organization account

Co-authored-by: Pccode66
Co-authored-by: pukkandan