Commit Graph

374 Commits (8fc54b1230e360fe6543bbbe8eea19ca2e8e06f3)

Author SHA1 Message Date
coletdjnz fe93e2c4cf
[youtube] misc cleanup and bug fixes ()
* Update some `_extract_response` calls to keep them consistent
* Cleanup continuation extraction related code using new API format
* Improve `_extract_account_syncid` to support multiple parameters
* Generalize `get_text` and related functions into one
* Update `INNERTUBE_CONTEXT_CLIENT_NAME` with integer values

Authored by: colethedj
coletdjnz 314ee30548
[youtube] Fix session index extraction and headers for non-web player clients ()
Fixes 
coletdjnz 34917076ad
[youtube] Fix authentication when using multiple accounts
`SESSION_INDEX` in `ytcfg` is the index of the active account and should be sent as `X-Goog-AuthUser` header

Closes 
Authored by @colethedj
The Hatsune Daishi ccc7795ca3
[yahoo:gyao:player] Relax `_VALID_URL` ()
Authored by: nao20010128nao
Felix S da1c94ee45
[generic] Extract previously missed subtitles ()
* [generic] Extract subtitles in cases missed previously
* [common] Detect discarded subtitles in SMIL manifests
* [generic] Extract everything in the SMIL manifest

Authored by: fstirlitz
coletdjnz 47193e0298
[youtube:tab] Extract playlist availability ()
Authored by: colethedj
coletdjnz 49bd8c66d3
[youtube:comments] Improve comment vote count parsing (fixes ) ()
Authored by: colethedj
Felix S 182b6ae8a6
[RTP] Fix extraction and add subtitles ()
Authored by: fstirlitz
felix 198f7ea89e
[extractor] Allow extracting multiple groups in `_search_regex`
From , Authored by: fstirlitz
coletdjnz c888ffb95a
[youtube] Use android client as default and add age-gate bypass for it ()
Authored by: colethedj
coletdjnz 9752433221
[youtube:comments] Fix `is_favorited` ()
Authored by colethedj
pukkandan f0ff9979c6
[vlive] Extract thumbnail directly in addition to the one from Naver
Closes 
coletdjnz 2d6659b9ea
[youtube:comments] Move comment extraction to new API ()
Closes , ,  

Authored by: colethedj
Kevin O'Connor c5370857b3
[BravoTV] Improve metadata extraction ()
Authored by: kevinoconnor7
pukkandan 6606817a86
[utils] Add `variadic`
zackmark29 73d829c144
[VIKI] Rewrite extractors ()
Closes 
Also added extractor-arg `video_types` to `vikichannel`

Co-authored-by: zackmark29, pukkandan
pukkandan 60bdb7bd9e
[youtube] Fix sorting of 3gp format
pukkandan 4bb6b02f93
Improve `extractor_args` parsing
pukkandan 51f8a31d65
Update to ytdl-commit-a803582
[peertube] only call description endpoint if necessary
a803582717
Tom-Oliver Heidel be05d5cff1
[soundcloud] Allow login using oauth token ()
Authored by: blackjack4494
zenerdi0de 30d569d2ac
[fancode] Fix extraction, support live and allow login with refresh token ()
Authored-by: zenerdi0de
OhMyBahGosh 08625e4125
[AdobePass] Add Spectrum MSO ()
From: https://github.com/ytdl-org/youtube-dl/pull/26792

Co-authored by: kevinoconnor7, ohmybahgosh
pukkandan 3acf6d3856
[Funimation] Rewrite extractor (See desc) ()
* Support direct `/player/` URL
* Treat the different versions of an episode as different formats of a single video. So `experience_id` can no longer be used as the video `id` and the `episode_id` is used instead. This means that all existing archives will break
* Extractor options `language` and `version` to pre-select them
* Compat option `seperate-video-versions` to fall back to old behavior (including using the old video IDs)

Closes 
pukkandan 46890374f7
[extractor] Minor improvements (See desc)
1. Allow removal of login hint - extractors can set their own login hint as part of `msg`
2. Cleanup `_merge_subtitles` signature
pukkandan 60755938b3
[extractor] Prevent unnecessary download of hls manifests
and refactor `hls_split_discontinuity` code
pukkandan bc97cdae67
[cleanup] Fix linter and some typos
Related: https://github.com/ytdl-org/youtube-dl/pull/29398
nyuszika7h e010672ab5
[videa] Fix extraction ()
Authored by: nyuszika7h
MinePlayersPE 17f0eb66b8
[RCTIPlus] Add extractor ()
Authored by: MinePlayersPE
pukkandan b1e60d1806 [facebook] Extract description and fix title
Partially fixes: 
krichbanana f6745c4980
[Youtube] Choose correct Live chat API for upcoming streams ()
Authored by: krichbanana
coletdjnz 109dd3b237
[youtube] Use new API for additional video extraction requests ()
Co-authored-by: colethedj, pukkandan
Closes https://github.com/yt-dlp/yt-dlp/issues/427
Workarounds for https://github.com/ytdl-org/youtube-dl/issues/29326, https://github.com/yt-dlp/yt-dlp/issues/319, https://github.com/ytdl-org/youtube-dl/issues/29086
LE 1e79316e20
[TBS] Support livestreams ()
Authored by: llacb47
coletdjnz 45261e063b
[youtube:comments] Fix error handling and add `itct` to params ()
Should close  (untested)

Authored by: colethedj
pukkandan 49c258e18d
[youtube] Fix subtitle names for age-gated videos
Related: https://github.com/iv-org/invidious/pull/2205#issuecomment-868680486
pukkandan 5d3a0e794b
Add `--extractor-args` to pass extractor-specific arguments
Mevious 125728b038
[funimation] Add `FunimationShowIE` ()
Closes 

Authored by: Mevious
LE 412cce82b0
[yahoo] Fix extraction ()
Fixes: https://github.com/ytdl-org/youtube-dl/issues/28290

Co-authored-by: llacb47, pukkandan
pukkandan 2b18a8c590
[plutotv] Improve `_VALID_URL`
Closes 
pukkandan ed807c1837
Update to ytdl-commit-379f52a
[liveleak] Remove extractor
379f52a495
Mevious 29f63c9672
[funimation] Extract subtitles ()
Closes , https://github.com/ytdl-org/youtube-dl/issues/25645
Related: https://github.com/ytdl-org/youtube-dl/pull/24906

Authored by: Mevious
pukkandan 9fc0de5796
[hotstar] Use server time for authentication instead of local time
Closes 
siikamiika c60ee3a218
[youtube_live_chat] Support ongoing live chat ()
Authored by: siikamiika
pukkandan e6779b9400
[twitcasting] Websocket support ()
Closes 
Authored by: nao20010128nao
pukkandan e36d50c5dd
[websockets] Add `WebSocketFragmentFD` ()
Necessary for 

Co-authored by: nao20010128nao, pukkandan
pukkandan 4e6767b5f2
[youtube] Temporary fix for age-gate
Related:
https://stackoverflow.com/a/67629882
https://github.com/yt-dlp/yt-dlp/issues/319
https://github.com/ytdl-org/youtube-dl/issues/29333
https://github.com/ytdl-org/youtube-dl/issues/29086
pukkandan d0fb4bd16f
[pornhub] Extract `cast`
Closes , https://github.com/ytdl-org/youtube-dl/pull/27384
felix 3fd4c2a543
[mediasite] Extract slides ()
Fixes:
https://github.com/ytdl-org/youtube-dl/issues/4974#issue-58006762
https://github.com/ytdl-org/youtube-dl/issues/4540#issuecomment-69574231
https://github.com/ytdl-org/youtube-dl/pull/11185#issuecomment-335554239

Authored by: fstirlitz
felix cdb19aa4c2
[downloader/mhtml] Add new downloader ()
This downloader is intended to be used for streams that consist of a
timed sequence of stand-alone images, such as slideshows or thumbnail
streams

This can be used for implementing:

https://github.com/ytdl-org/youtube-dl/issues/4974#issue-58006762
https://github.com/ytdl-org/youtube-dl/issues/4540#issuecomment-69574231
https://github.com/ytdl-org/youtube-dl/pull/11185#issuecomment-335554239

https://github.com/ytdl-org/youtube-dl/issues/9868
https://github.com/ytdl-org/youtube-dl/pull/14951


Authored by: fstirlitz
pukkandan 8326b00aab
Allow `images` formats
Necessary for .

* They are identified by `vcodec=acodec='none'`
* These formats show as the worst in `-F`
* Any postprocessor that expects audio/video will be skipped
* `b*` and all related selectors will skip such formats
* This commit also does not add any selector for downloading such formats. They have to be explicitly requested by the `format_id`. Implementation of a selector is left for when  is resolved
pukkandan 45db527fa6
[youtube] Login is not needed for `:ytrec`
coletdjnz 4ba001080f
[youtube] Non-fatal alert reporting for unavailable videos page ()
Co-Authored by: colethedj, pukkandan
coletdjnz 1974e99f4b
[youtube] Improve SAPISID cookie handling (closes ) ()
Author: colethedj
pukkandan 3b1fe47d84
Release 2021.06.08
pukkandan 46358f647d
Update to ytdl-commit-c2350ca
Update MSVC 2010 redist URL
c2350cac24
pukkandan 89ee4cf8ae
[viki] Fix extraction
Closes 
Code from: 59e583f7e8
pukkandan eb0f9d6838
[zoom] Extract transcripts as subtitles
pukkandan d3d8d8184a
[extractor] Fix pre-checking archive for some extractors
The `id` regex group must be present for `_match_id` and pre-checking archive to work correctly
pukkandan e85a39717a
[twitcasting] Add TwitCastingUserIE, TwitCastingLiveIE
Closes 

Code adapted from: f1fb9222bb/youtube_dl/extractor/twitcasting.py
Authored by: pukkandan, nao20010128nao
MinePlayersPE f2cd7060fc
[vidio] Add VidioPremierIE and VidioLiveIE ()
Authored-by: MinePlayersPE
felix e2efe599aa
[common] Fix FourCC fallback when parsing ISM ()
In some DASH manifests, the FourCC attribute is actually present,
but empty.  We thus apply the same fallback to 'AACL' that we do
when the attribute is entirely absent.

Authored by: fstirlitz
pukkandan 5e1dba8ed6
Remove duplicate file `trovolive.py`
pukkandan bea742222f
[youtube] Support shorts URL
Closes 
Ashish 6e6390321c
[Hotstar] Add HotStarSeriesIE ()
Authored by: Ashish0804
pukkandan 14eb1ee1cb
Update to ytdl-commit-d495292
[ard] Relax _VALID_URL and fix video ids
d495292852

Closes 
coletdjnz 879e7199bb
[archiveorg] Add YoutubeWebArchiveIE ()
Co-authored by: colethedj, pukkandan, alex-gedeon
MinePlayersPE 10bb7e51e8
[vidio] Add login support ()
Authored by: MinePlayersPE
pukkandan b0089e8992
[fancode] Add extractor (#316,#354)
Closes , 

Authored by: rmsmachine
Ashish a3ed14cbaf
[Voot] Add VootSeriesIE ()
Authored by: Ashish0804
LE 9dee4df559
[Saitosan] Add new extractor ()
Closes 
Authored by: llacb47
pukkandan adddc50cbf
[extractor] Functions to parse socket.io response as json
Authored by: pukkandan, llacb47
MinePlayersPE 46c43ffc9d
[vidio] Support premium videos ()
Authored by: MinePlayersPE
pukkandan 37a3bb66a7
[extractor] Allow `note=False` when extracting manifests
pukkandan 46953e7e6e
[youtube:playlist] fix bug
pukkandan 077c476276
[zee5] Fix m3u8 formats extension
rhsmachine 65af1839c6
[patreon] Support vimeo embeds ()
Authored by: rhsmachine
pukkandan 177877c544
[extractor] Always prefer native hls downloader by default
When the manifest is not downloadable by native downloader, it already is able to detect it and switch to `ffmpeg`. So there doesn't seem to be a reason anymore to use ffmpeg as the preferred downloader
Oliver Freyermuth f17c702270
[ard] Allow URLs without `-` before id
https://github.com/ytdl-org/youtube-dl/pull/29091

Authored by: olifre
pukkandan 3907333c5d
[extractor] Skip subtitles without URI in m3u8 manifests
Closes 

Authored by: hheimbuerger
Ashish 09d18ad07e
[Sonyliv] Add subtitle support ()
Authored by: Ashish0804
Ashish 9572eaaa11
[ShemarooMe] Add extractor ()
Closes 
Co-authored-by: Ashish0804, pukkandan
pukkandan 8d68ab98a7
[youtube] Fix bug where not all hls formats were extracted
Bug introduced in 9297939ec3
Ashish 135e6b93f4
[SonyLIV] Add SonyLIVSeriesIE ()
Authored by: Ashish0804
king-millez 13a49340ed
[telemundo] add extractor ()
Closes 
Authored by: king-millez
pukkandan 09f1580e2d
[youtube] `/live` URLs should raise error if channel is not live
Fixes: https://github.com/ytdl-org/youtube-dl/issues/29090
pukkandan b73612a254
Update to ytdl-commit-dfbbe29
[redbulltv] fix embed data extraction
dfbbe2902f
king-millez 5014558ab9
[parlview] Add extractor ()
Authored by: king-millez
coletdjnz 00ae27690d
[youtube] Add `html5=1` param to `get_video_info` page requests ()
Workaround for , https://github.com/ytdl-org/youtube-dl/issues/29086
Authored by: colethedj
pukkandan 9d5d4d64f8
[youtube] Better message when login required
king-millez 98784ef8d6
[audius:artist] Add extractor ()
Authored by: king-millez
pukkandan d3fc8074a4
[youtube] Sort audio-only formats correctly
Closes 
pukkandan 07cce701de
[cleanup] linter, code formatting and readme
king-millez 74e001af1d
[tenplay] Fix extractor ()
Authored by: king-millez
pukkandan ff2751ac9c
[youtube] Always extract `maxresdefault` thumbnail
Fixes: https://github.com/ytdl-org/youtube-dl/issues/29049
pukkandan abcdd12b26
[youtube:tab] Support youtube music `MP` pages
pukkandan 18db754858
[youtube:tab] Redirect `UC` channels that doesn't have a `videos` tab
Many topic URLs don't have a videos tab, but has an equivalent `UU` playlist.
If there is no playlist, fallback to using channel page
pukkandan fe03a6cdc8
[youtube:tab] Support youtube music `VL` and `browse` pages
pukkandan cd684175ad
[youtube:tab] Support channel search
Fixes: https://github.com/ytdl-org/youtube-dl/issues/29071
pukkandan da692b7920
[cleanup] youtube tests
pukkandan 95c01b6c16
[youtube:tab] Show alerts only from the final webpage
pukkandan a06916d98e
[extractor] Add `write_debug` and `get_param`
coletdjnz 9297939ec3
[Youtube] Extract more formats for `music.youtube` URLs ()
Based on: https://github.com/ytdl-org/youtube-dl/pull/28778, https://github.com/ytdl-org/youtube-dl/pull/26160

Co-authored-by: craftingmod, colethedj, pukkandan
pukkandan 774d79cc4c
[youtube] Add language names
Co-authored by: nixxo, tpikonen
Based on: https://github.com/ytdl-org/youtube-dl/pull/26112
Closes: 
pukkandan 2412044c90
Add field `name` for subtitles
Co-authored by: pukkandan, tpikonen

Based on: , https://github.com/ytdl-org/youtube-dl/pull/26112
pukkandan 120916dac2
[youtube] multiple subtitles in same language
Fixes: https://github.com/ytdl-org/youtube-dl/issues/21164
Related: , https://github.com/ytdl-org/youtube-dl/pull/26112
pukkandan 0fb983f62d
[youtube] Extract audio language
pukkandan 0760b0a7e2
Standardize `write_debug`
pukkandan d908aa636a
[cleanup] Fix typos
pukkandan 3d89341b47
[common] bugfix for when `compat_opts` is not given
pukkandan d8ec40b39f
[rmcdecouverte] Generalize `_VALID_URL`
Closes 
pukkandan fac988053f
Release 2021.05.11
* and some documentation improvements
pukkandan 61241abbb0
[generic] Respect the encoding in manifest
pukkandan 53ed7066ab
Option `--compat-options` to revert some of yt-dlp's changes
* Deprecates `--list-formats-as-table`, `--list-formats-old`
pukkandan 2f567473c6
[Plugins] Prioritize plugins over standard extractors
and prevent plugins from overwriting the standard extractor classes

Closes 
pukkandan 41d1cca328
Update to ytdl-commit-a726009
[blinkx] Remove extractor
a726009987
pukkandan 12e73423f1
[plutotv] Fix format extraction for some urls
* And fallback to the first urls if ad-free urls can't be found
Closes 
pukkandan 7700b37f39
[plutotv] Extract subtitles from manifests
Ashish c28cfda81f
[SonyLiv] Fix `title` and `series` extraction ()
Authored by: Ashish0804
pukkandan 3158150cb7
[utils] Add `network_exceptions`
pukkandan 06425e9621
[blinkx] Minor fix
Fixes: https://github.com/ytdl-org/youtube-dl/issues/28941
pukkandan 1815d1028b
[zee5] Fix py2 compatibility
The Hatsune Daishi 0fa9a1e236
[whowatch] Add extractor
closes 

Authored by: nao20010128nao 
Modified from: 9e4a0e061a/youtube_dl/extractor/whowatch.py
pukkandan 80185155a1
[ukcolumn] Add Extractor
Closes 
pukkandan c755f1901f
[CBS] Improve `_VALID_URL` to support movies
Closes 
Tested by: BeeMuffins
pukkandan 68b91dc905
[youtube] Add `oembed` to reserved names
pukkandan 88f06afc0c
[rmcdecouverte] Improve `_VALID_URL`
Closes 
CXwudi 40078a55e2
[niconico] Fix bug in thumbnail extraction
Bug from: 6b1d8c1e30
Authored by: CXwudi
pukkandan 07e4a40a9a
[crackle] Improve extraction (See desc)
Closes 

* Refactor authorization as an extension to `_download_json`
* Better error messages and warnings
* Respect `--ignore-no-formats-error`
* Extract subtitles from manifests
* Try with crackle's geo-location service if all hard-coded countries fail
pukkandan e28f1c0ae8
[cleanup] Fix linter and some typos
* Also remove inconsistent use of `"` in setup.py
pukkandan ef39f8600a
[curiositystream] Fix collections
Closes 

* A bug with authentication was reported in <https://github.com/yt-dlp/yt-dlp/issues/277#issuecomment-828254721> but cannot be tested without an account
pukkandan 2291dbce2a
[niconico] Fix HLS formats
Closes 

* The structure of the API JSON was changed
* Smile Video seems to be no longer available. So remove the warning
* Move ping to downloader
* Change heartbeat interval to 40sec
* Remove unnecessary API headers

Authored-by: CXwudi, tsukumijima, nao20010128nao, pukkandan
Tested by: tsukumijima
pukkandan 895b0931e5
[youtube:tab] Detect playlists inside community posts
pukkandan 1ad047d0f7
[nebula] Move to nebula.app
Closes 
Tested by: Lamieur
pukkandan be6202f12b
Subtitle extraction from streaming media manifests
Authored by fstirlitz
Modified from: https://github.com/ytdl-org/youtube-dl/pull/6144

Closes: 
Fixes:
https://github.com/ytdl-org/youtube-dl/issues/6106
https://github.com/ytdl-org/youtube-dl/issues/14977
https://github.com/ytdl-org/youtube-dl/issues/21438
https://github.com/ytdl-org/youtube-dl/issues/23609
https://github.com/ytdl-org/youtube-dl/issues/28132

Might also fix (untested):
https://github.com/ytdl-org/youtube-dl/issues/15424
https://github.com/ytdl-org/youtube-dl/issues/18267
https://github.com/ytdl-org/youtube-dl/issues/23899
https://github.com/ytdl-org/youtube-dl/issues/24375
https://github.com/ytdl-org/youtube-dl/issues/24595
https://github.com/ytdl-org/youtube-dl/issues/27899

Related:
https://github.com/ytdl-org/youtube-dl/issues/22379
https://github.com/ytdl-org/youtube-dl/pull/24517
https://github.com/ytdl-org/youtube-dl/pull/24886
https://github.com/ytdl-org/youtube-dl/pull/27215

Notes:
* The functions `extractor.common._extract_..._formats` are still kept for compatibility
* Only some extractors have currently been moved to using `_extract_..._formats_and_subtitles`
* Direct subtitle manifests (without a master) are not supported and are wrongly identified as containing video formats
* AES support is untested
* The fragmented TTML subtitles extracted from DASH/ISM are valid, but are unsupported by `ffmpeg` and most video players
    * Their XML fragments can be dumped using `ffmpeg -i in.mp4 -f data -map 0 -c copy out.ttml`.
        Once the unnecessary headers are stripped out of this, it becomes a valid self-contained ttml file
    * The ttml subs downloaded from DASH manifests can also be directly opened with <https://github.com/SubtitleEdit>
* Fragmented WebVTT files extracted from DASH/ISM are also unsupported by most tools
    * Unlike the ttml files, the XML fragments of these cannot be dumped using `ffmpeg`
    * The webtt subs extracted from DASH can be parsed by <https://github.com/gpac/gpac>
    * But validity of the those extracted from ISM are untested
Felix S e8f834cd8d [threeqsdn] Extract subtitles from streaming manifests
Felix S e0e624ca7f [canvas] Extract subtitles from streaming manifests
Felix S ec4f374c05 [wat] Extract subtitles from streaming manifests
Felix S c811e8d8bd [atresplayer] Extract subtitles from streaming manifests
Felix S b2cd5da460 [francetv] Extract subtitles from the HLS manifest
Felix S 2de3b21e05 [uplynk] Extract subtitles from HLS manifests
Felix S 4bed436371 [twitter] Extract subtitles from HLS manifests
Felix S efe9dba595 [srgssr] Extract subtitles from HLS manifests
Felix S 47f4203dd3 [nytimes] Extract subtitles from HLS manifests
Felix S 015c10aeec [roosterteeth] Use common code for subtitle extraction
Felix S a00d781b73 [elonet] Use common code for subtitle extraction
Felix S 0c541b563f [tv4] Extract subtitles from streaming manifests
Felix S 64a5cf7929 [byutv] Extract subtitles from streaming manifests
Felix S 7a450a3b1c [generic] Extract subtitles from direct SSTR manifest links
Felix S 7de27caf16 [generic] Extract subtitles from direct DASH manifest links
Felix S c26326c1be [generic] Extract subtitles from direct HLS manifest links
Felix S 66a1b8643a [downloader/ism] Support muxing TTML subtitles