Commit Graph

12495 Commits (0efbc6b56d2b030e5dc98fa7f533a2e6cd41cf30)

Author SHA1 Message Date
Sergey M․ da42ff0668
[iprima] Improve geo restriction detection and disable geo bypass
Sergey M․ 82f662182b
[iprima] Modernize
Sergey M․ 2cc7fcd338
[commonmistakes] Disable UnicodeBOM extractor test for python 3.2
Sergey M․ 6d4c259765
[svt] PEP 8
Sergey M․ c78dd35491
[nrk] PEP 8
Sergey M․ 8ffb8e63fe
[prosiebensat1] Throw ExtractionError on unsupported page type (closes )
Sergey M․ 983e9b7746
[nrk] Update _API_HOST and relax _VALID_URL
Sergey M․ c58b7ffef4
[tv4] Bypass geo restriction and improve detection
Sergey M․ f1a78ee4ef
[tv4] Switch to hls3 protocol (closes )
Sergey M․ de64e23c56
[downloader/ism] Honor HTTP headers when downloading fragments
Sergey M․ 553f6dbac7
[downloader/dash] Honor HTTP headers when downloading fragments
For example, https://www.oppetarkiv.se/video/1196142/natten-ar-dagens-mor
Sergey M․ 0aa10994f4 [options] Move geo restriction related options to separate section
Sergey M․ 4248dad92b Improve geo bypass mechanism
* Rename options to preffixly match with --geo-verification-proxy
* Introduce _GEO_COUNTRIES for extractors
* Implement faking IP right away for sites with known geo restriction
Sergey M․ 0a840f584c Rename bypass geo restriction options
Sergey M․ 0016b84e16 Add faked X-Forwarded-For to formats' HTTP headers
Sergey M․ 18a0defab0 [utils] Make random_ipv4 return unicode string
Sergey M․ 5d3fbf77d9 [viki] Improve geo restriction detection
Sergey M․ 80b59020e0 [vgtv] Improve geo restriction detection
Sergey M․ 71631862f4 [srgssr] Improve geo restriction detection
Sergey M․ 89cc7fe770 [vbox7] Improve geo restriction detection and use geo bypass mechanism
Sergey M․ 04d906eae3 [svt] Improve geo restriction detection and use geo bypass mechanism
Sergey M․ 8ab8066cf0 [pbs] Improve geo restriction detection and use geo bypass mechanism
Sergey M․ 01b1aa9ff4 [ondemandkorea] Improve geo restriction detection and use geo bypass mechanism
Sergey M․ ff4007891f [nrk] Improve geo restriction detection and use geo bypass mechanism
Sergey M․ 28200e654b [itv] Improve geo restriction detection and use geo bypass mechanism
Sergey M․ e633f21a96 [go] Improve geo restriction detection and use geo bypass mechanism
Sergey M․ d392005a79 [dramafever] Improve geo restriction detection and use geo bypass mechanism
Sergey M․ 773f291dcb Add experimental geo restriction bypass mechanism
Based on faking X-Forwarded-For HTTP header
Pierre Mdawar bf5b9d859a [utils] Introduce YoutubeDLError base class for all youtube-dl exceptions
Remita Amine 049a0f4d6d [brightcove:legacy] restrict videoPlayer value(closes )
Sergey M․ ac33accd96
[options] Mention quoted string literals for --match-filter
Sergey M․ e84888b432
[tvn24] Improve extraction (closes )
Jakub Wilk 02d9b82a23
[tvn24] Add extractor
Sergey M․ a2e3286676
[thisav] Add support for html5 media (closes )
Sergey M․ f75caf059e
[metacafe] Improve (closes )
Alex Monk bdabbc220c
[metacafe] Bypass family filter
If you don't send this user=ffilter: false cookie, it will 301 redirect you to a page asking about it, and then the title check will fail.
Remita Amine 70bcc444a9 [viceland] improve info extraction and update test
Sergey M․ 28e35f5070
release 2017.02.17
Sergey M․ 2c1f442c2b
[options] Add missing spaces
Sergey M․ bad4ccdb5d
[heise] Improve (closes )
Tobias Gruetzmacher db76c30c6e
[heise] Support videos embedded in any article.
Déstin Reed c2bde5d081
[ellentv] Improve
Sergey M․ 90fad0e74c
[openload] Fix extraction (closes )
Vijay Singh d94badc755
[openload] Semifix extraction (closes )
just updated the code. i don't do much python still i tried to convert my code. lemme know if there is any prob with it
Yen Chi Hsuan fef51645d6
[theplatform] Recognize URLs with whitespaces (closes )
Sergey M․ 4cead6a614
[einthusan] Relax _VALID_URL (closes , closes )
Yen Chi Hsuan a4a554a793
[generic] Try parsing JWPlayer embedded videos (closes )
Sergey M․ b898f0a173
[elpais] Fix typo and improve extraction (closes )
Sergey M․ 2480b056c1
release 2017.02.16
Sergey M․ eafaeb226a
[ceskatelevize] Lower priority for audio description sources ()
Sergey M․ de4d378c0c
[ceskatelevize] Prefix format ids
Sergey M․ db13c16ef8
[utils] Add support for quoted string literals in --match-filter (closes , closes , closes )
Remita Amine 1bd05345ea [amcnetworks] fix extraction(closes )
Sergey M․ 3021cf83b7
[pinkbike] Fix uploader extraction (closes )
Sergey M․ 04a741232f
[onetpl] Add support for businessinsider.com.pl and plejada.pl
Sergey M․ 43a3d9edfc
[onetpl] Add support for onet.pl (closes )
Sergey M․ d31aa74fdb
[onetmvp] Add shortcut extractor
Sergey M․ 6092ccd058
[vodpl] Make more robust and add another test (closes )
Marek Rusinowski 22ce9ad2bd
[vod.pl] Add new extractor
Sergey M․ 9a372f14b4
[pornhub] Extract video URL from tv platform site (, )
Sergey M․ 5cb2d36c82
[ceskatelevize] Extract DASH formats (closes , closes )
Ondřej Caletka fcca0d53a8
[ceskatelevize] Quick fix to revert to using old HLS-based playlist
This fixes recent changes in iVysilani. Proper patch should migrate to
MPEG-DASH version, which is now the default.
Sergey M․ 58a65ba852
release 2017.02.14
Sergey M․ 50de3dbad3
[zdf] Fix extraction (closes )
Sergey M․ 085f169ffe
[xtube] Fix extraction for both kinds of video id (closes )
Vobe f6d6ca1db3
[xtube] Improve title extraction
Sergey M․ 6e5956e6ba
[lemonde] Fallback delegate extraction to generic extractor (closes , closes )
Remita Amine 89c6691f9d [bellmedia] accept longer video id(closes )
Remita Amine 454e5cdb17 [limelight] add support referer protected videos
Remita Amine 9dad941853 [disney] improve extraction
- add support for more urls
- detect expired videos
- skip Adobe Flash Access protected videos

closes 
closes 
closes 
closes 
Remita Amine 0dac7cbb09 [hotstar] improve extraction(closes )
- extract all qualities
- detect drm protected videos
- extract more metadata
Yen Chi Hsuan f8514630db
[einthusan] Fix extraction (closes )
The old test URLs are no longer valid, so I replace them with the one
from 
Aniruddh-J 459818e280 [aenetworks] Add support for lifetimemovieclub.com
Sergey M․ 6310acf512
[youtube] Fix parsing codecs (closes )
Yen Chi Hsuan f3915452de Merge pull request from wiiaboo/python2
utils.py: Workaround TypeError with Python 2.7.13 in Windows
Ricardo Constantino 2f49bcd690
utils.py: Workaround TypeError with Python 2.7.13 in Windows
Fixes 

Tested with Windows Python 2.7.12 and 2.7.13.
Yen Chi Hsuan 68c22c4c15
[iqiyi] Update _TESTS
Sergey M․ 9b92a5917b
release 2017.02.11
Sergey M․ 3d7e3aaa0e
[pluralsight:course] Fix extraction (closes )
Sergey M․ 624c4b92ff
[facebook] Add coding cookie
Thomas Christlieb 2af12ad9d2 Introduce get_elements_by_class and get_elements_by_attribute utility functions
Remita Amine 97eb9bd2ac [bbc] extract m3u8 formats with 320k audio
Sergey M․ 71cdd75628
[facebook] Relax video id matching (closes , closes , closes )
Remita Amine c7d6f614f3 [corus] Add new extractor(closes )()
Remita Amine 08a00eef79 [extractor/common] skip m3u8 manifests protected with Adobe Flash Access
Sergey M․ 9dd5408c99
[pluralsight] Detect blocked account error message ()
Sergey M․ 9510709575
[bloomberg] Add another video id regex (closes )
Remita Amine 5abcca9060 [sixplay] use raw string for regex
Sergey M․ e01bfc19c3
[extractor/commonmistakes] Restrict _VALID_URL (closes )
Remita Amine 4d32b63851 [tvplayer] Add new extractor
Sergey M․ 55d4de2283
release 2017.02.10
Sergey M․ ff24261ba0
[kaltura] Add explicit port to regexes
They should not match e.g. cdnapi.kaltura.computernetworks.com/...
Sergey M․ fbc6dc525e
[xtube] Fix shortcuts
Sergey M․ 9150d1eb69
[xtube] Fix extraction (closes )
Sergey M․ b7f9843bec
[pornhub] Simplify (closes )
Thomas Christlieb e64b0fca14
[pornhub] Fix extraction (closes )
Sergey M․ 78ef214d2d
[facebook] Improve JS data regex (closes )
Remita Amine be670b8e8f [external:ffmpeg] do not assume that ffmpeg unknown version format is new
Remita Amine 37084f6641 [kaltura] improve embed partner id extraction(fixes )
Remita Amine b04975733c [sprout] Add new extractor
Remita Amine c8b8fb0a99 [sixplay] improve extraction
- skip drm protected formats
- extract more and better formats
- skip duplicate asset urls
Remita Amine 8298018273 [scrippsnetworks:watch] Add new extractor(closes )
Remita Amine ae8d5a5c59 [go] add support for adobe pass auth(closes )(closes )
Sergey M․ b9c9cb5f79
[6play] Fix extraction (closes )
Remita Amine fdf9b959bc [nbc] add support adobe pass auth(closes )
Sergey M․ 013877298d
release 2017.02.07
Sergey M․ f28aeff264
[pornhub] Fix extraction (closes )
Sergey M․ 242a14a1f6
[extractor/common] Fix audio only with audio group in m3u8 (closes )
Sergey M․ d5d904ff7d
[canalplus] Add support for cstar.fr ()
Sergey M․ 5620f840f6
[extractor/generic] Add test for and more metadata for rtmp
Sergey M․ b7a8c1bcfa
[extractor/generic] Improve rtmp support (closes )
Sergey M․ 7097bffba6
[downloader/fragment] Respect --no-part
Sergey M․ 2aec7256ae
[extractor/common] Speed-up media tags regex (closes )
Yen Chi Hsuan 9c14fe9681
[gaskrank] Minor change and update ChangeLog after
motophil e705755739 [gaskrank] Add new extractor ()
* [gaskrank] Add new extractor

* [gaskrank] Add new extractor - fixes as requested

* [gaskrank] Add new extractor - style fix

* [Gaskrank] Add new extractor - requested fixes

* [Gaskrank] Add new extractor - fix md5 checksum

* [gaskrank] Add new extractor - more requested fixes

* [Gaskrank] Add new extractor - fixed all but one quantified code issues

* [Gaskrank] add new extractor - more fields extracted, added second test

* [Gaskrank] Add new extractor - requested fixes.

* [Gaskrank] Add new extractor - requested changes.

* [Gaskrank] Add new extractor - final(?) fixes.
Yen Chi Hsuan 019f4c0371
[bandcamp] Fix extraction for incomplete albums
Closes 
Yen Chi Hsuan 2ab2c0d1f5
[iwara] Add width (closes )
The heuristic is from 
Yen Chi Hsuan caf0f5f8b7
[iwara] Fix extraction (closes )
Yen Chi Hsuan e4e50f60b1
[googledrive] Fix extraction on Python 3.6
Since Python 3.6, invalid escape sequences are deprecated. It's likely
that there are invalid escape sequences somewhere on the webpage, so
instead of unescaping the whole webpage, just unescape the URL.

See https://bugs.python.org/issue27364. That change was designed for
string literals, while it affects the 'unicode_escape' encoding as well.
The code path is:

str.decode('unicode_escape')
    codecs.unicode_escape_decode()
        PyUnicode_DecodeUnicodeEscape()
Sergey M․ 6ef3e65a7b
[videopress] Add extractor
Sergey M․ 6fd138bed8
[sportbox] PEP 8
Remita Amine 3d2c2752c5 [afreecatv] extract rtmp formats
Sergey M․ a713a86755
release 2017.02.04.1
Sergey M․ 9db8f6c540
[twitch:stream] Improve _VALID_URL (closes )
Remita Amine 8e4041cf3f [radiocanada] fix extraction for toutv rtmp formats
Sergey M․ 31487eb974
release 2017.02.04
John Hawkinson c2521c1ac6 [Piksel] Add another app token regex
A Connecticut Princess 643dc0fcfe
[vk] Catch author blocked error message
Example link (video in blocked group):
https://vk.com/search?c%5Bq%5D=%D0%9F%D1%80%D1%8B%D0%B6%D0%BE%D0%BA%20c%20%D0%BA%D1%80%D0%B0%D0%BD%D0%B0%20%D0%B2%20%D1%81%D1%82%D0%B8%D0%BB%D0%B5%20%D0%A7%D0%B5%D0%BB%D0%BE%D0%B2%D0%B5%D0%BA%D0%B0-%D0%BF%D0%B0%D1%83%D0%BA%D0%B0&c%5Bsection%5D=video&c%5Bsort%5D=2&z=video-10639516_456240611
Remita Amine 36fce54816 [turner] fix downloading of secure hls formats using ffmpeg(closes )(closes )(closes )
Remita Amine 2c15db829c [drtv] add support for live and radio sections(closes )(closes )
Remita Amine f65dba7cdb [myspace] fix extraction and extract hls and http formats
Remita Amine 605fd6392f [youtube] add format info for itag 325 and 328
Sergey M․ f962790ee5
[vine] Fix extraction (closes )
Sergey M․ b7cc5f078e
[extractors] Remove remnants of sportbox extractor ()
Sergey M․ f7a10d8cd6
[sportbox] Remove extractor (closes )
Covered by generic extractor
Yen Chi Hsuan 8939f784d9 Merge pull request from ThomasChr/randonplaylistorder
New parameter --playlist-random to randomize playlist download order. Fixes 
Remita Amine df0588a31f Merge branch 'fstirlitz-filmon'
Remita Amine 4ce3407d08 [filmon] improve extraction
Mattias Wadman 45024183ae [infoq] Add audio only format if available ()
* [infoq] Add audio only format if available

Refactor cookie code into a function.
Renamed formats to http_video, http_audio, rtmp_video
Renamed extract functions to video instead of videos as they return
one or no video.

* [infoq] Rename to _extract_cookies as it more than one

* [infoq] Remove redundant determine_ext

* [infoq] Add comment about hardcoded URL

* [infoq] Use _hidden_inputs instead of messy regex

* [infoq] Probe if audio URL is valid

Make it possible to pass headers to _is_valid_url

* [infoq] Add audio only test
Justsoos 33da98f493
[douyutv] Improve room id regex
http://www.douyu.com/t/lpl  source get extra '\' with "room_id\" (from js coding)
Sergey M․ 4195096ea8
[utils] Improve comments processing in js_to_json (closes )
Michal Čihař 0bbcc8a10a
[iprima] Fix extraction (closes , closes )
Michal Čihař b3ee552e4b
[utils] Handle single-line comments in js_to_json
Yen Chi Hsuan a22b2fd19b
[youtube] Fix ytsearch* when cookies are provided
Closes 

The API with `page` is no longer used in browsers, and YouTube always
returns {'reload': 'now'} when cookies are provided.

See http://youtube.github.io/spfjs/documentation/start/ for how SPF
works. Basically appending static link with a `spf` parameter yields the
corresponding dynamic link.
Sergey M․ c54c01f82d
[go] Relax video id regex (closes )
Sergey M․ 5a116e1302
[facebook] Fix title extraction (closes )
Sergey M․ a685751051
[youtube:playlist] Recognize TL playlists (closes )
Yen Chi Hsuan bd8f48c78b
[bilibili] Support new Bangumi URLs (closes )
To reduce complexity, I don't support old Bangumi URLs directly via
_VALID_URL. Instead, I choose to let it go to generic redirection. An
example can be found in :

http://bangumi.bilibili.com/anime/v/40062
Remita Amine 81aeafeb44 [cbc:watch] extract audio codec for audion only formats(fixes )
Remita Amine 8bdc149441 [downloader/external:ffmpeg] minimize the use of aac_adtstoasc filter
Jaime Marquínez Ferrándiz 020c5df52d [elpais] Fix extraction for some URLs (closes )
Remita Amine da162c1135 [compat] add compat_etree_register_namespace to __all__ list
Sergey M․ 5069594993
release 2017.02.01
Sergey M․ b83ef507b4
[facebook] Fix extraction (closes )
Yen Chi Hsuan fe5aa197b5
[prosiebensat1] PEP8 and update _TESTS
Thomas Christlieb 7882f1115e Added new Regex for prosiebensat1 Extractor Description. Fixes ()
* Added new Regex for prosiebensat1 Extractor Description. Fixes 

* Using _og_search_description() as a Fallback for Description-Regex

* Using _og_search_description() as a Fallback for Description-Regex - Second try

* Also added fallback regex

* Using _og_search_description() as a Fallback for Description-Regex - Third try

* removed fatal=False from search for description regex. default=None should be preferred only

* Using fatal=false for _og_search_description

* Revert "Using fatal=false for _og_search_description"

This reverts commit 2b7e123f9d0f2bd6ada54fa8e4e6035fece5dbf4.

* Deleted default=None Parameter for _og_search_property
Yen Chi Hsuan 2b2d5d319b
[crunchyroll] Remove ScaledBorderAndShadow settings
See https://github.com/rg3/youtube-dl/pull/9028, especially @lachs0r's
comments for the reason behind this change
Sergey M․ 26c0f09935
[vimeo] PEP 8
Sergey M․ c15cd29640
[vimeo] Extract upload timestamp
Sergey M․ c38a67bcd5
[vimeo] Extract license (closes )
Sergey M․ 7c5329e6f4
[nrk] Improve extraction and update tests (closes )
Odd Stråbø 8fd65faece
[NRKTV] Added NRKTVSeriesIE
[NRKTV] Added season and episode number to metadata.

[NRKTV] Added category to metadata.

[NRKTV] Added tests to NRKTVSeries.

[NRKTV] Fixed whitespace issues (flake8).
Sergey M․ d7e215b42d
release 2017.01.31
Sergey M․ 3c90cc8b6f
[youtube] Fix extraction for domainless player URLs
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Remita Amine ae9a173b64 [vimeo] extract both mixed and separated dash formats
Thomas Christlieb 75822ca790 New parameter --playlist-random to randomize playlist download order. Fixes
Remita Amine dadb836139 [ruutu] extract dash formats
Remita Amine 4719419951 [itv] fix extraction in python 2.6
Remita Amine c2d9c25f81 [compat] add compat_etree_register_namespace
Sergey M․ 4d2fdb07c4
release 2017.01.29
Alex Seiler f13da8af28
[azmedien:playlist] Add support for topic and themen playlists
Sergey M․ e228616c6e
[extractor/common] Fix initialization template (closes , closes )
Sergey M․ c58c2d63cb
[extractor/common] Document forgotten fragment base and path interfaces
Sergey M․ d04621daf4
[extractor/common] Fix duration per dash segment (closes )
Andre Walker 56fc078da8 [npo] Update subtitles url
NPO websites changed the domain they used for subtitles, from
e.omroep.nl to tt888.omroep.nl.
Sergey M․ 1a241a2d02
[options] Refactor autonumber options and add validation (closes , closes , closes )
Costy Petrisor acbb2374bc
added --autonumber-start NUMBER as a command line option to be able to offset the index at which autonumber formats filenames
Remita Amine 4edeac5bfa [itv] fix subtitle extraction
Remita Amine f592ff9868 [itv] extract subtitles
Sergey M․ 24ee6b9721
[options] Remove experimental mark from some options
Remita Amine a71b8d3b3b [itv] Add new extractor(closes )
Sergey M․ 732fb3f8be
[options] Move --abort-on-unavailable-fragment to download section
Sergey M․ 008f247077
[mtv81] Add extractor (closes )
Sergey M․ 661cc229d2
[vlive:channel] Improve
ping b92d3c5343 [vlive] Add support for channels
Sergey M․ ab6f6aee78
[kaltura] Add fallback for fileExt
Sergey M․ 26e40542dd
[kaltura] Improve uploader_id extraction
Alex Seiler 99a0baf370
[konserthusetplay] Add support for rspoplay.se
Sergey M․ d41ed6d243
release 2017.01.28
Sergey M․ e0b6e50ccd
[crunchyroll] Improve series and season metadata extraction (closes )
Sergey M․ 3a194cb4ec
[soundcloud] Improve formats extraction and extract audio bitrate
Sergey M․ 9b73471801
[soundcloud] Extract hls formats
Sergey M․ 489ffc1182
[soundcloud] Fix track URL extraction (closes )
Sergey M․ 0b23c222ba
[twitch:vod] Expand _VALID_URL (closes )
Remita Amine b51a4ebed4 [aenetworks] fix season episodes extraction(fixes )
Remita Amine 9463637887 [tva] Add new extractor(closes )
Sergey M․ 3cbecdd111
[jamendo] Improve and extract more metadata (closes )
Sergey M․ 15846398ca
[utils] Improve parse_duration
Stefan Pöschel c19ef77c31
[jamendo] Extract full title
Remita Amine b3277115a1 [disney] Add new extractor(closes )(closes )()()