Commit Graph

572 Commits (7d740e7dc7149cfd93dde1fa47e9f314e72582c2)

Author SHA1 Message Date
Sergey M․ b52c9ef165
[extractor/generic] Improve support for pornhub embeds (closes )
Sergey M․ e28ed498e6
[extractor/generic] Add support for redtube embds (closes )
Sergey M․ 37e7a71c6c
[extractor/generic] Add support for drtuber embds (closes )
Sergey M․ 4119a96ce5
[extractor/generic] Skip URLs we came from when delegating ISM extraction
Sergey M․ 26aae56690
[extractor/generic] Improve ISM extraction
Sergey M․ cc99a77ac1
[extractor/generic] Add support for ISM manifests
Sergey M․ 773017c648
[rentv] Move rentv test from generic extractor and add only matching tests
Yen Chi Hsuan 9dcd6fd3aa
[generic,commonprotocols] Move mms suuport from GenericIE
And use _generic_* helpers in those extractors
Yen Chi Hsuan 3d83a1ae92
[generic] Support direct MMS links (closes )
Sergey M․ 09b9c45e24
[generic] Add support for multiple vimeo embeds (Closes )
Déstin Reed dcdb292fdd
Unify coding cookie
Yen Chi Hsuan bd26441205
[utils] Fix xattr error handling
Sergey M․ 86f4d14f81
Refactor fragments interface and dash segments downloader
- Eliminate segment_urls and initialization_url
+ Introduce manifest_url (manifest may contain unfragmented data in this case url will be used for direct media URL and manifest_url for manifest itself correspondingly)
* Rewrite dashsegments downloader to use fragments data
* Improve generic mpd extraction
Remita Amine d05ef09d9d [mangomolo] fix domain regex
Remita Amine 7d273a387a [mangomolo] add support for Mangomolo embeds
Sergey M․ a5ff05df1a
[extractor/generic] Add vimeo embed that requires Referer passed
Remita Amine 93b8404599 [generic,vodplatform] improve embed regex
Yen Chi Hsuan 39efc6e3e0 [generic] Update some _TESTS
Sergey M․ fd3ec986a4
[generic] Fix dbtv test (Closes )
Déstin Reed b0c8f2e9c8
[DBTV:generic] Add support for embeds
Sergey M․ 2a1321a272
[vbox7:generic] Add support for vbox7 embeds
Sergey M․ 082395d0a0
[extractor/generic] Add proper default to _search_json_ld call
Sergey M․ 395c74615c
Revert "[extractor/generic] Make _search_json_ld non fatal"
This reverts commit 958849275f.
Sergey M․ 958849275f
[extractor/generic] Make _search_json_ld non fatal
Remita Amine 217d5ae013 [vodplatform] Add new extractor
Sergey M․ db19df6ca0
[extractor/generic] Add test for
Sergey M․ 94aae01548
[extractor/generic] Extract all soundcloud embeds (Closes )
Yen Chi Hsuan 326fa4e6e5
[generic] Skip an invalid test
Yen Chi Hsuan 05d1e7aaa9
[generic] Fix an MTV test and another test that breaks nosetests
Sergey M․ 1979969f91
[extractor/generic] Add support for arkena embeds
Sergey M․ 371ddb14fe
[extractor/generic] Change twitter:player embeds priority to lowest (Closes )
Yen Chi Hsuan c03adf90bd
[generic] Add the test. Closes
Sergey M․ 6e6b70d65f
[extractor/generic] Properly comment out a test
Sergey M․ 0de168f7ed
[extractor/generic] Detect schema.org/VideoObject embeds
Yen Chi Hsuan ad213a1d74
[francetv] Recognize more Dailymotion embedded videos
Closes 
Remita Amine 81953d1ae5 [kaltura] add support videos stored on custom kaltura servers(closes )
Yen Chi Hsuan dbf0157a26 [generic] Add MD5 checksums
Yen Chi Hsuan 7deef1ba67 [generic] Support Wordpress "YouTube Video Importer" plugin
Closes 
Yen Chi Hsuan fd6ca38262 [facebook] Improve Facebook embedded detection
Related to .

Another example comes from 9834872bf6.
Remita Amine 7dbeee7e22 [generic] make twitter:player extraction non fatal
Remita Amine 329179073b [generic] add generic support for twitter:player embeds
Sergey M․ 06a96da15b
[eagleplatform] Improve embed detection and extract in separate routine (Closes )
Sergey M․ c287f2bc60
[extractor/generic] Use _extract_url for kaltura embeds (Closes )
Sergey M․ 427cd050a3
[extractor/generic] Improve kaltura embed detection (Closes )
Yen Chi Hsuan 7d52c052ef
[generic] Fix test_Generic_76
Broken: https://travis-ci.org/rg3/youtube-dl/jobs/140251658
Yen Chi Hsuan c5edd147d1
[generic] Remove an invalid test
Now handled by telewebion.py
Yen Chi Hsuan 1fa309da40
[generic] Update test_Generic_40
The original link now redirects to an YouTube user channel.
Yen Chi Hsuan cc4444662c [generic] Remove Vulture embed detection
Vulture.com videos now hosts on YouTube, Vimeo, MTV, NBC News or Hulu.
Here's an example of Hulu:
http://www.vulture.com/2016/06/kimmel-interviews-mariah-carey-in-a-bathtub.html
Yen Chi Hsuan de3eb07ed6
[generic] Detect NBC News embeds
Sergey M․ 48a5eabc48
[extractor/generic] Add support vessel embeds (Closes )
Yen Chi Hsuan 811586ebcf
[generic] Update the UDNEmbed test case
Yen Chi Hsuan 28bab13348
[generic,viewlift] Move a test case to the specialized extractor
Yen Chi Hsuan 877032314f
[generic] Improve Kaltura detection
Closes 
Yen Chi Hsuan 78d3b3e213
[generic] Improve Livestream detection (closes )
Yen Chi Hsuan 4a12077855
[genric] Eliminate duplicated video URLs (closes )
Sergey M․ 4d8819d249
[extractor/generic] Add support for theplatform embeds (Closes , closes )
Sergey M․ 6c114b1210
[extractor/generic] Remove generic id and title from wistia extractionand update tests
Sergey M․ 7ded6545ed
[extractor/generic] Add test for wistia standard embed
Sergey M․ aa5957ac49
[extractor/generic] Add support for async wistia embeds (Closes )
Sergey M․ 6f41b2bcf1
[extractor/generic] Improve 3qsdn embeds support (Closes )
Sergey M․ 5d39176f6d
[extractor/generic:3qsdn] Add support for embeds
remitamine 67167920db [viewlift] replace SnagFilms extractors
- add support for other sites that use the same logic
- improve format extraction and sorting
Yen Chi Hsuan 4645432d7a
[eagleplatform] Checking direct HTTP links
Sometimes they fail with 404
Yen Chi Hsuan 0738187f9b
[ThePlatform] Fix tests failed since 79ba9140dc
Yen Chi Hsuan 949b6497cc
[generic] Unescape the video URL
Fixes 
Sergey M․ 11e60fcad8
[extractor/generic] Improve instagram embeds (Closes )
Yen Chi Hsuan ef0e4e7bc0
[generic] Fix test_Generic_2
Now a HEAD request returns 400 Bad Request
Kacper Michajłow b8f67449ec [generic] Add support for LiveLeak embeds
Yen Chi Hsuan 134c207e3f
[arte.tv:embed] Extended support ()
remitamine f9b1529af8 [generic] remove sbnation test(handled by VoxMediaIE)
Sergey M․ 19dbaeece3 Remove _sort_formats from _extract_*_formats methods
Now _sort_formats should be called explicitly.
_sort_formats has been added to all the necessary places in code.

Closes 
Yen Chi Hsuan d10fe8358c [generic] Add a test case for brightcove embed
Closes 
Yen Chi Hsuan 5a51775a58 [generic] Extract Instagram embeds ()
Yen Chi Hsuan 088e1aac59 [generic] Support Vine embeds ()
Sergey M․ 0d769bcb78 [extractor/generic] Fix missing byte literal prefix
Sergey M․ edd9b71c2c [extractor/generic] Add a test for m3u playlist served without proper Content-Type
Sergey M․ 5940862d5a [extractor/generic] Detect m3u playlists served without proper Content-Type
Sergey M․ de6c51e88e [extractor/generic] Fix direct link semantics
Sergey M․ 303dcdb995 [extractor/generic] Simplify upload_date extraction
Sergey M․ 20938f768b [extractor/generic] Add another test for generic m3u8
Sergey M․ 955737b2d4 [extractor/generic] Force Content-Type to lowecase
Sergey M․ 263eff9537 [extractor/generic] Properly extract format id from Content-Type
Fixes extraction for cases like: audio/x-mpegURL; charset=utf-8
Sergey M․ 3f2f4a94aa [extractor/generic] Extract f4m formats from final URLs
Sergey M․ f930e0c76e [extractor/generic] Extract f4m formats and refactor common info
Sergey M․ 2c9ca78281 [extractor/generic] Add support for tnaflix network embeds (Closes )
Sergey M․ 33d4fdabfa [extractor/generic] Add support for ok embeds ()
Sergey M․ 9d939cec48 [extractor/generic] Add direct mpd url test
Sergey M․ 7be747b921 [extractor/generic] Pass mpd base url to _parse_mpd_formats
Sergey M․ 5a4905924d [extractor/generic] Improve dailymotion embed detection (Closes , closes )
remitamine 28ad7df65d [generic] detect MPD manfiest only from the content
Sergey M․ 79a3508579 [extractor/generic] Detect DASH manifests in found URLs and extract mpd formats
Sergey M․ 1b840245bd [extractor/generic] Detect DASH manifests and extract mpd formats
remitamine bcbbb98bfe [generic] extract dash formats detected using content type
remitamine eadc3ccd50 [generic] extract m3u8 formats when mpegurl content type detected
remitamine 1bf996fa5c [generic] Add support for Limelight API
Sergey M․ aecfcd4e59 [ultimedia] Rename to digiteka
Sergey M․ 1fa2b9841d [extractor/generic] Extend dailymotion embed regex
Jakub Wilk dfb1b1468c Fix typos
Closes .
Sergey M․ ff18735cb2 [extractor/generic] Add support for videomore embeds
remitamine 06e4874c99 Merge branch 'jukebox' of https://github.com/remitamine/youtube-dl into remitamine-jukebox
remitamine 7ba71e30fb Merge branch 'bliptv' of github.com:remitamine/youtube-dl into remitamine-bliptv
remitamine 7cb0952474 [makertv] improve extraction
remitamine a8ae232fa9 Merge branch 'googledrive' of github.com:remitamine/youtube-dl into remitamine-googledrive
remitamine 5b251628e9 [googledrive] Modernize
Sergey M․ 45dad7ba1b [extractor/generic] Use _extract_url for pladform
remitamine 53e06b2507 [ooyala] fix duration scale
remitamine c3d3110f6a Merge pull request from remitamine/ooyala
[ooyala] extract more formats and metadata
remitamine 78653a33aa Merge remote-tracking branch 'upstream/master' into bliptv
Sergey M? 5c2266df4b Switch codebase to use sanitized_Request instead of
compat_urllib_request.Request

[downloader/dash] Use sanitized_Request

[downloader/http] Use sanitized_Request

[atresplayer] Use sanitized_Request

[bambuser] Use sanitized_Request

[bliptv] Use sanitized_Request

[brightcove] Use sanitized_Request

[cbs] Use sanitized_Request

[ceskatelevize] Use sanitized_Request

[collegerama] Use sanitized_Request

[extractor/common] Use sanitized_Request

[crunchyroll] Use sanitized_Request

[dailymotion] Use sanitized_Request

[dcn] Use sanitized_Request

[dramafever] Use sanitized_Request

[dumpert] Use sanitized_Request

[eitb] Use sanitized_Request

[escapist] Use sanitized_Request

[everyonesmixtape] Use sanitized_Request

[extremetube] Use sanitized_Request

[facebook] Use sanitized_Request

[fc2] Use sanitized_Request

[flickr] Use sanitized_Request

[4tube] Use sanitized_Request

[gdcvault] Use sanitized_Request

[extractor/generic] Use sanitized_Request

[hearthisat] Use sanitized_Request

[hotnewhiphop] Use sanitized_Request

[hypem] Use sanitized_Request

[iprima] Use sanitized_Request

[ivi] Use sanitized_Request

[keezmovies] Use sanitized_Request

[letv] Use sanitized_Request

[lynda] Use sanitized_Request

[metacafe] Use sanitized_Request

[minhateca] Use sanitized_Request

[miomio] Use sanitized_Request

[meovideo] Use sanitized_Request

[mofosex] Use sanitized_Request

[moniker] Use sanitized_Request

[mooshare] Use sanitized_Request

[movieclips] Use sanitized_Request

[mtv] Use sanitized_Request

[myvideo] Use sanitized_Request

[neteasemusic] Use sanitized_Request

[nfb] Use sanitized_Request

[niconico] Use sanitized_Request

[noco] Use sanitized_Request

[nosvideo] Use sanitized_Request

[novamov] Use sanitized_Request

[nowness] Use sanitized_Request

[nuvid] Use sanitized_Request

[played] Use sanitized_Request

[pluralsight] Use sanitized_Request

[pornhub] Use sanitized_Request

[pornotube] Use sanitized_Request

[primesharetv] Use sanitized_Request

[promptfile] Use sanitized_Request

[qqmusic] Use sanitized_Request

[rtve] Use sanitized_Request

[safari] Use sanitized_Request

[sandia] Use sanitized_Request

[shared] Use sanitized_Request

[sharesix] Use sanitized_Request

[sina] Use sanitized_Request

[smotri] Use sanitized_Request

[sohu] Use sanitized_Request

[spankwire] Use sanitized_Request

[sportdeutschland] Use sanitized_Request

[streamcloud] Use sanitized_Request

[streamcz] Use sanitized_Request

[tapely] Use sanitized_Request

[tube8] Use sanitized_Request

[tubitv] Use sanitized_Request

[twitch] Use sanitized_Request

[twitter] Use sanitized_Request

[udemy] Use sanitized_Request

[vbox7] Use sanitized_Request

[veoh] Use sanitized_Request

[vessel] Use sanitized_Request

[vevo] Use sanitized_Request

[viddler] Use sanitized_Request

[videomega] Use sanitized_Request

[viewvster] Use sanitized_Request

[viki] Use sanitized_Request

[vk] Use sanitized_Request

[vodlocker] Use sanitized_Request

[voicerepublic] Use sanitized_Request

[wistia] Use sanitized_Request

[xfileshare] Use sanitized_Request

[xtube] Use sanitized_Request

[xvideos] Use sanitized_Request

[yandexmusic] Use sanitized_Request

[youku] Use sanitized_Request

[youporn] Use sanitized_Request

[youtube] Use sanitized_Request

[patreon] Use sanitized_Request

[extractor/common] Remove unused import

[nfb] PEP 8
Yen Chi Hsuan 750b9ff032 [generic] Extract M3U8 formats (closes )
Yen Chi Hsuan 28602e747c [generic] Refactor
Yen Chi Hsuan 6cc37c69e2 [generic] Unescape URLs from JWPlayer ()
Sergey M․ 6da620de58 [kaltura] Add test for referrer protected video ()
Sergey M․ 5b5fae5f20 [generic] Use referrer from source kaltura embed URLs ()
Yen Chi Hsuan c39fd7b1ca [UDNEmbed] Fix generic UDN pages
Closes 
Sergey M․ 3b7d9aa487 Rename all references to legacy studio Brightcove extractor
Sergey M․ 1f4b722b00 [generic] Clarify Brightcove Legacy Studio comment
Sergey M․ f6519f89b0 [generic] Extract Brightcove New Studio embeds
Sergey M․ 5c17f0a67a [brightcove:embedinpage] Rename extractor to brightcove new
It's not actually embed_in_page but "New Studio" and allows both iframe and embed_in_page embeds
Sergey M․ 4fcaa4f4a5 [brightcove] Rename extractor to brightcove legacy
Old embedding approaches are now "Legacy Studio"
Sergey M․ a2973eb597 Merge branch 'brightcove_in_page_embed' of https://github.com/remitamine/youtube-dl into remitamine-brightcove_in_page_embed
Sergey M․ 6a5d6de1e3 [generic] Improve kaltura embed detection (2)
Sergey M․ 1c31a5b0e0 [generic] Improve kaltura embed detection (Closes )
Sergey M 30eecc6a04 Merge pull request from jaimeMF/xml_attrib_unicode
Use a wrapper around xml.etree.ElementTree.fromstring in python 2.x (…
Sergey M․ e327b736ca [generic] Update test
Jaime Marquínez Ferrándiz f78546272c [compat] compat_etree_fromstring: also decode the text attribute
Deletes parse_xml from utils, because it also does it.
remitamine cce9d15d01 [ooyala] extract domain,handle errors and change related tests
remitamine 77302fe5c9 [bliptv] remove extractor and add support for site replacement(makertv)
Sergey M․ dd467d33d0 [extractor/generic] Add support for condenast script embeds (Closes , closes )
Sergey M․ 46fde8a1a2 [extractor/generic] Use _extract_url for mtvservices
remitamine 6aeba407db [jukebox] remove extractor and handle it using generic extractor
Sergey M․ 6c91a5a7f5 [extractor/generic] Fix following redirect in Refresh HTTP header on python 2
remitamine ed1269000f [brightcove] add support for brightcove in page embed(fixes )
felix 12439dd5ec [generic] Capture <audio> tags in addition to <video> tags
Sergey M․ 1f8125805e [generic] Add test for screenwavemedia embed
Sergey M․ efd712c69b [generic] Use screenwavemedia embed pattern
felix 8ca31a0e05 [generic] Look for ScreenwaveMedia embeds
Sergey M․ 0791ac1b44 [extractor/generic] Clarify comment
Sergey M․ 1de5cd3ba5 [extractor/generic] Add test for xspf playlist
Sergey M․ 729accb482 [extractor/generic] Add support for xspf playlists
Sergey M. d5d7bdaeb5 Merge pull request from dstftw/improve-generic-smil-support
Improve generic SMIL support
Sergey M․ bf94d763ba [extractor/generic] Add test for
Sergey M․ 8a37aa1517 [extractor/generic] Expand ooyala regex (Closes )
Sergey M․ 54a9328b20 [generic] Expand jwplayer support
Sergey M․ 8765222d22 [extractor/generic] Add generic SMIL tests
Sergey M․ e5e8d20a3a [extractor/generic] Improve generic SMIL detection
Sergey M․ 88ed52aec9 [bbc] Add support for direct bbc.co.uk embeds
remitamine 653789afc7 add google drive embeds
Jaime Marquínez Ferrándiz b37317d8b0 [generic] Unescape HTML escape sequences in redirect urls (fixes )
Yen Chi Hsuan a1b85269a4 [extractor/generic] Support vid.me embeds
Sergey M․ cbd55ade68 [extractor/generic] Add support for francetv embeds
Sergey M․ f7e6f7fa23 [extractor/generic] Use compat_urllib_parse_unquote
fnord 45eedbe58c Generic: use compat_urllib_parse_unquote to prevent utf8 mangling
of the entire page in python 2.

-requires- fixed compat_urllib_parse_unquote

example - the following will save with a mangled playlist title,
 instead of the kanji for 'tsunami'. This affects all utf8encoded
 urls as well

youtube-dl -f18 -o '%(playlist_title)s-%(title)s.%(ext)s' \
  61c14c1e3a/tsunami.html
Sergey M․ bf20b9c540 [extractor/generic] Add test for myvi embed
Sergey M․ 6dd94d3a79 [myvi:embed] Rename to myvi
Sergey M․ e6c2d9ad29 [extractor/generic:myvi] Add support for myvi embeds
Sergey M․ 66e568de3b [extractor/generic] Improve kaltura embeds support (Closes )
Sergey M․ b8c1cc1a51 [extractor/generic] Add test for snagfilms embeds
Sergey M․ eedd20ef96 [extractor/generic] Add support for snagfilms embeds
Sergey M․ 8084be78c5 [extractor/generic] Add test for OnionStudios embeds
Sergey M․ 1ac1c4c26e [extractor/generic] Add support for OnionStudios embeds (Closes )
Sergey M. 3c3b4176bd Merge pull request from dstftw/force-generic-extractor
Add --force-generic-extractor
Sergey M․ 2fece970b8 [extractor/generic] Clarify generic extraction warning
Yen Chi Hsuan a5158f38a3 [generic/adobetv] Support AdobeTVVideo embeds ()
Sergey M․ c76799c555 [extractor/generic] Add test for xhamster embed
Sergey M․ 2bb5b6d0a1 [generic] Add support for xhamster embeds
Yen Chi Hsuan 756f574e4e [dailymotion/generic] Add DailymotionCloudIE
Yen Chi Hsuan b407e173e4 [vimeo/generic] Move detection logic from GenericIE to VimeoIE
Sergey M․ 78e2b74bb9 [tumblr] Add support for pornhub embeds (Closes )
Sergey M․ 65d161c480 [extractor/generic] Add support for pornhub embeds
Sergey M․ d22dec74ff Add `--force-generic-extractor`
For some extractors that are hard to workout a good _VALID_URL we use very vague and unrestrictive ones,
e.g. just allowing anything after hostname and capturing part of URL as id.
If some of these extractors happen to have an video embed of some different hoster or platform
and this scenario was not handled in extractor itself we end up with inability to download this embed
until extractor is fixed to support embed of this kind.
Forcing downloader to use the generic extractor can be a neat temporary solution for this problem.
Example: FiveTV extractor with Tvigle embed - http://www.5-tv.ru/rabota/broadcasts/48/
Sergey M․ 9872d3110c [extractor/generic] Add support for tvigle embeds
Sergey M․ b859971873 [extractor/generic] Rename tvc embed url variable
Sergey M․ f37bdbe537 [extractor/generic] Add test for tvc embed
Sergey M․ 2da09ff8b0 [extractor/generic] Fix tvc ie_key
Sergey M․ 954c1d0529 [tvc] Refactor extractor names
Sergey M․ 494f20cbdc [extractor/generic] Add support for tvc embeds
Sergey M․ 97b570a94c [generic] Improve rtl.nl embeds detection (Closes )
Yen Chi Hsuan b26733ba7f [brightcove] Allow single quotes in Brightcove URLs (fixes )
Sergey M․ c5138a7ce4 [extractor/generic] Clarify test comment
Sergey M․ c5fa81fe81 [extractor/generic] Put all direct link tests near to each other for better navigation
Sergey M․ a074e92296 [extractor/generic] Add test for large compressed media
Sergey M․ 1ddb9456c4 [extractor/generic] Use compat_urllib_parse_unquote for unquoting video_id and title from URL
Sergey M․ 58bde34a23 [extractor/generic] Force Accept-Encoding to any for extraction pass
Sergey M․ 339516072b [extractor/generic] Unescape video_id and title extracted from URL
Sergey M․ 25c3a7348f [generic] Fix typo
Sergey M․ b827a6015c [generic] Add test for sportbox embeds
Sergey M․ d40a3b5b55 [generic] Add support for sportbox embeds
Jaime Marquínez Ferrándiz 3b5f65a64c [mlb] Fix extraction of articles
And move test from generic, since it's directly handled by MLBIE
Sergey M․ dc455a5f88 [extractor/generic] Add test for svt embed
Sergey M․ bab19a8e91 [extractor/generic] Add support for svt embeds (Closes )
Sergey M․ 406224be52 [extractor/generic] Fix following incomplete redirects ()
Yen Chi Hsuan 8001607e90 [generic] Detect more MLB videos (fixes )
Yen Chi Hsuan 0954cd8aa4 [Cinemassacre] Add detection for videos from blip.tv
Yen Chi Hsuan 2fe1b5bd2a [CSpan] Add detection for Senate ISVP. Closes
Sergey M․ 6225984681 [generic] Update pladform embed test
Yen Chi Hsuan d6fd958c5f [generic] Extract videos from SMIL manifests (closes and fixes )
Sergey M․ c798f15b98 [generic] Add test for playwire embed ()
Sergey M․ 2dcc114f84 [generic] Add support for playwire embeds (Closes )
Yen Chi Hsuan 01c58f8473 [generic] Fix test generic_51
The website replaced the original video with a new one