Commit Graph

1265 Commits (bcec568ea7556d122b79bf802b1ac3ff814531f6)

Author SHA1 Message Date
Kenshin bcec568ea7 Add ffmpeg progress tracking to FFmpegFD
Add ffmpeg progress tracking to FFmpegPostProcessor

Apply changes from the code review

Fix a bug where the subprocess didn't capture any output thus an empty stdout and stderr were sent back

Add missing hooks

Revert "Add missing hooks"

This reverts commit a359c5ea10bb35b965e80801e736f43cdbcf3294.

Add support of -ss=132 timestamp format

Infer filename from ffmpeg args instead of info_dic

Remove redundant parenthesis and switch from to_stodout to to_screen

Add info kwarg with multiple files and ffmpeg to track progress

Moved format progress function to util

Moved format progress function to util

Add progress tracking to postprocessing operations

Fix typing error

Handle self._downloader is None at __init__

Move format progress functions to utils

Move format progress functions to utils

Handle case where ydl passed is None

Handle case where ydl passed is None

Handle case where _multiline isn't initialized

Handle case where _multiline isn't initialized

Fix streams incorrectly returned

Fix case where ydl is nested in the downloader

Add progress_hook attribute

Fix bug after merge

Fix import bugs after merge

Catch up with upstream

Fix merge errors #1

Adapt tests and implementatation for ffmpeg progress tracking args
1 year ago
Simon Sawicki 4e38e2ae9d
[rh:requests] Handle both `bytes` and `int` for `IncompleteRead.partial` (Fix 8a8b54523a) (#8348)
Authored by: bashonly, coletdjnz, Grub4K
2 years ago
coletdjnz 8a8b54523a
[rh:requests] Add handler for `requests` HTTP library (#3668)
Adds support for HTTPS proxies and persistent connections (keep-alive)

Closes https://github.com/yt-dlp/yt-dlp/issues/1890
Resolves https://github.com/yt-dlp/yt-dlp/issues/4070
Resolves https://github.com/ytdl-org/youtube-dl/issues/32549
Resolves https://github.com/ytdl-org/youtube-dl/issues/14523
Resolves https://github.com/ytdl-org/youtube-dl/issues/13734

Authored by: coletdjnz, Grub4K, bashonly
2 years ago
Awal Garg 9d7ded6419
[utils] `js_to_json`: Fix `Date` constructor parsing (#8295)
Authored by: awalgarg, Grub4K
2 years ago
Simon Sawicki 088add9567
[cleanup] Misc
Authored by: Grub4K
2 years ago
Simon Sawicki de015e9307
[core] Prevent RCE when using `--exec` with `%q` (CVE-2023-40581)
The shell escape function is now using `""` instead of `\"`. `utils.Popen` has been patched to properly quote commands.

Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg for reference.

Authored by: Grub4K
2 years ago
Simon Sawicki 61bdf15fc7
[core] Raise minimum recommended Python version to 3.8 (#8183)
Authored by: Grub4K
2 years ago
bashonly 5ca095cbcd
[cleanup] Misc (#8182)
Closes #7796, Closes #8028
Authored by: barsnick, sqrtNOT, gamer191, coletdjnz, Grub4K, bashonly
2 years ago
bashonly 661c9a1d02
[test:download] Test for `expected_exception`
Authored by: at-wat

Co-authored-by: Atsushi Watanabe <atsushi.w@ieee.org>
2 years ago
bashonly 52414d64ca
[utils] `js_to_json`: Handle `Array` objects
Authored by: Grub4K, std-move

Co-authored-by: std-move <26625259+std-move@users.noreply.github.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
2 years ago
coletdjnz 20fbbd9249
[networking] Fix various socks proxy bugs (#8065)
- Fixed support for IPv6 socks proxies
- Fixed support for IPv6 over socks5
- Fixed --source-address not being obeyed for socks4 and socks5
- Fixed socks4a when the destination address is an IPv4 address

Closes https://github.com/yt-dlp/yt-dlp/issues/7959
Fixes https://github.com/ytdl-org/youtube-dl/issues/15368

Authored by: coletdjnz
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
Co-authored-by: bashonly <bashonly@bashonly.com>
2 years ago
coletdjnz fcd6a76adc
[tests] Add tests for socks proxies (#7908)
Authored by: coletdjnz
2 years ago
coletdjnz db7b054a61
[networking] Add request handler preference framework (#7603)
Preference functions that take a request and a request handler instance can be registered to prioritize different request handlers per request.

Authored by: coletdjnz
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
2 years ago
pukkandan 6148833f5c
[cleanup] Misc 2 years ago
pukkandan 3f7965105d
[utils] HTTPHeaderDict: Handle byte values 2 years ago
pukkandan de20687ee6
[test] Fix `test_load_certifi`
Closes #7688, #7675
2 years ago
coletdjnz 4bf912282a
[networking] Remove dot segments during URL normalization (#7662)
This implements RFC3986 5.2.4 remove_dot_segments during the URL normalization process.

Closes #3355, #6526

Authored by: coletdjnz
2 years ago
coletdjnz bbeacff7fc
[networking] Ignore invalid proxies in env (#7704)
Authored by: coletdjnz
2 years ago
coletdjnz 95abea9a03
[test] Fix `httplib_validation_errors` test for old Python versions (#7677)
Fixes https://github.com/yt-dlp/yt-dlp/issues/7674

Authored by: coletdjnz
2 years ago
coletdjnz 86aea0d3a2
[networking] Add strict Request extension checking (#7604)
Authored by: coletdjnz
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
2 years ago
pukkandan a250b24733
[compat] Ensure submodules are imported correctly
Closes #7663
2 years ago
pukkandan 62b5c94cad
[cleanup] Misc fixes
Closes #7528
2 years ago
pukkandan a264433c9f
[outtmpl] Fix replacement for `playlist_index` 2 years ago
bashonly e57eb98222
[fd/external] Fix ffmpeg input from stdin (#7655)
Bugfix for 1ceb657bdd

Authored by: bashonly
2 years ago
bashonly af86873218
[utils] Improve `parse_duration`
Authored by: bashonly
2 years ago
bashonly 75dc8e673b
[networking] Fix `--legacy-server-connect` (#7645)
Bugfix for 227bf1a33b

Authored by: bashonly
2 years ago
bashonly 71baa490eb
[networking] Fix POST requests with zero-length payloads (#7648)
Bugfix for 227bf1a33b

Authored by: bashonly
2 years ago
bashonly 42ded0a429
[fd/external] Fixes to cookie handling
- Fix bug in `axel` Cookie header arg
- Pass cookies to `curl` as strings
- Write session cookies for `aria2c` and `wget`

Closes #7539
Authored by: bashonly
2 years ago
bashonly 6c5211cebe
[core] Fix HTTP headers and cookie handling
- Remove `Cookie` header from `http_headers` immediately after loading into cookiejar
- Restore compat for `--load-info-json` cookies
- Add more tests
- Fix improper passing of Cookie header by `MailRu` extractor

Closes #7558
Authored by: bashonly, pukkandan
2 years ago
coletdjnz 3d2623a898
[compat, networking] Deprecate old functions (#2861)
Authored by: coletdjnz, pukkandan
2 years ago
coletdjnz 227bf1a33b
[networking] Rewrite architecture (#2861)
New networking interface consists of a `RequestDirector` that directs
each `Request` to appropriate `RequestHandler` and returns the
`Response` or raises `RequestError`. The handlers define adapters to
transform its internal Request/Response/Errors to our interfaces.

User-facing changes:
- Fix issues with per request proxies on redirects for urllib
- Support for `ALL_PROXY` environment variable for proxy setting
- Support for `socks5h` proxy
   - Closes https://github.com/yt-dlp/yt-dlp/issues/6325, https://github.com/ytdl-org/youtube-dl/issues/22618, https://github.com/ytdl-org/youtube-dl/pull/28093
- Raise error when using `https` proxy instead of silently converting it to `http`

Authored by: coletdjnz
2 years ago
pukkandan c365dba843
[networking] Add module (#2861)
No actual changes - code is only moved around
2 years ago
pukkandan 1b392f905d
[utils] Add temporary shim for logging
Related: #5680, #7517
2 years ago
Mahmoud Abdel-Fattah 2af4eeb772
[utils] `clean_podcast_url`: Handle more trackers (#7556)
Authored by: mabdelfattah, bashonly
Closes #7544
2 years ago
Simon Sawicki 3121512228
[core] Change how `Cookie` headers are handled
Cookies are now saved and loaded under `cookies` key in the info dict
instead of `http_headers.Cookie`. Cookies passed in headers are
auto-scoped to the input URLs with a warning.

Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: Grub4K
2 years ago
coletdjnz f8b4bcc0a7
[core] Prevent `Cookie` leaks on HTTP redirect
Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: coletdjnz
2 years ago
bashonly 1ceb657bdd
[fd/external] Scope cookies
- ffmpeg: Calculate cookies from cookiejar and pass with `-cookies` arg instead of `-headers`
- aria2c, curl, wget: Write cookiejar to file and use external FD built-in cookiejar support
- httpie: Calculate cookies from cookiejar instead of `http_headers`
- axel: Calculate cookies from cookiejar and disable http redirection if cookies are passed
    - May break redirects, but axel simply don't have proper cookie support

Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: bashonly, coletdjnz
2 years ago
pukkandan 47bcd43724
[outtmpl] Pad `playlist_index` etc even when with internal formatting
Closes #7501
2 years ago
pukkandan cd810afe2a
[extractor/youtube] Improve nsig function name extraction 2 years ago
pukkandan 42f2d40b47
Update to ytdl-commit-07af47
[YouTube] Improve fix for ae8ba2c
07af47960f
2 years ago
pukkandan 1619ab3e67
Bugfix for ebe1b4e34f 2 years ago
pukkandan ad54c9130e
[cleanup] Misc
Closes #6288, Closes #7197, Closes #7265, Closes #7353, Closes #5773
Authored by: mikf, freezboltz, pukkandan
2 years ago
pukkandan ebe1b4e34f
[outtmpl] Fix some minor bugs
Closes #7164
2 years ago
pukkandan eedda5252c
[utils] `FormatSorter`: Improve `size` and `br`
Closes #1596

Previously, when some formats have accurate size and some approximate,
the ones with accurate size was always prioritized

For formats with known tbr and unknown vbr/abr, we were setting
(vbr=tbr, abr=0) for sorting to work. This is no longer needed.

Authored by pukkandan, u-spec-png
2 years ago
pukkandan 01aba2519a
[jsinterp] Fix global object extraction
Closes #7327
2 years ago
bashonly b4a252fba8
[jsinterp] Fix division (#7279)
* Fixes nsig decryption for Youtube JS player `8c7583ff`

Authored by: bashonly
2 years ago
Matt Broadway b38d4c941d
[cookies] Update for chromium changes (#6897)
Authored by: mbway
2 years ago
coletdjnz daafbf49b3
[core] Support decoding multiple content encodings (#7142)
Authored by: coletdjnz
2 years ago
coletdjnz b87e01c123
[cookies] Move `YoutubeDLCookieJar` to cookies module (#7091)
Authored by: coletdjnz
2 years ago
coletdjnz 08916a49c7
[core] Improve HTTP redirect handling (#7094)
Aligns HTTP redirect handling with what browsers commonly do and RFC standards. 

Fixes issues afac4caa7d missed.

Authored by: coletdjnz
2 years ago