Commit Graph

549 Commits (90634acfcf9b106c9f5ac12bae5d92307cd577b0)

Author SHA1 Message Date
Random User c73e330e7a _find_jwplayer_data() returns dict or None
This simplifies code for callers of `_find_jwplayer_data()` which no longer have
to run `_parse_json()` on the return value.

It also makes sure that `_find_jwplayer_data()` returns either a `dict` or
`None` and nothing else.
8 years ago
John Hawkinson 46b18f2349 [BostonGlobe] New. Nonstandard version of Brightcove.
Has a "data-brightcove-video-id" instead of a "data-video-id," otherwise
pretty much just Brightcove. Except the Globe isn't all Brightcove
videos, so fallback to Generic, too.

Also, abstract playlist_from_matches() from generic.py to common.py, and use
it here.

History of these changes can be found in
51170427d4b1143572a498dedaee61863a5b2c5b.
8 years ago
Sergey M․ b51dc9db0e
[extractor/common] Extract SMIL formats from jwplayer 8 years ago
Sergey M․ 1a2192cb90
[extractor/common] Pass arguments to _parse_jwplayer_formats and PEP8 8 years ago
Sergey M․ 0236cd0dfd
[extractor/common] Improve height extraction and extract bitrate 8 years ago
Sergey M․ ed0cf9b383
[extractor/common] Move jwplayer formats extraction in separate method 8 years ago
Yen Chi Hsuan eeb0a95684
[extractor/common] Add 'preference' to _parse_html5_media_entries
Some websites, like NJPWorld, put different qualities on different
player pages.
8 years ago
Sergey M․ eea0716cae
[extractor/common] Print origin country for fake IP 8 years ago
Sergey M․ 336a76551b
[extractor/common] Do not quit _initialize_geo_bypass on empty countries 8 years ago
Sergey M․ dc0a869e5e
[extractor/common] Fix typo 8 years ago
Sergey M․ e39b5d4ab8
[extractor/common] Allow calling _initialize_geo_bypass from extractors (#11970) 8 years ago
Sergey M․ 3ccdde8cb7
[extractor/common] Emphasize geo bypass APIs are experimental 8 years ago
Sergey M․ 4248dad92b Improve geo bypass mechanism
* Rename options to preffixly match with --geo-verification-proxy
* Introduce _GEO_COUNTRIES for extractors
* Implement faking IP right away for sites with known geo restriction
8 years ago
Sergey M․ 0a840f584c Rename bypass geo restriction options 8 years ago
Sergey M․ 0016b84e16 Add faked X-Forwarded-For to formats' HTTP headers 8 years ago
Sergey M․ 773f291dcb Add experimental geo restriction bypass mechanism
Based on faking X-Forwarded-For HTTP header
8 years ago
Yen Chi Hsuan a4a554a793
[generic] Try parsing JWPlayer embedded videos (closes #12030) 8 years ago
Remita Amine 08a00eef79 [extractor/common] skip m3u8 manifests protected with Adobe Flash Access 8 years ago
Sergey M․ 242a14a1f6
[extractor/common] Fix audio only with audio group in m3u8 (closes #11995) 8 years ago
Sergey M․ 2aec7256ae
[extractor/common] Speed-up media tags regex (closes #11979) 8 years ago
Mattias Wadman 45024183ae [infoq] Add audio only format if available (#11565)
* [infoq] Add audio only format if available

Refactor cookie code into a function.
Renamed formats to http_video, http_audio, rtmp_video
Renamed extract functions to video instead of videos as they return
one or no video.

* [infoq] Rename to _extract_cookies as it more than one

* [infoq] Remove redundant determine_ext

* [infoq] Add comment about hardcoded URL

* [infoq] Use _hidden_inputs instead of messy regex

* [infoq] Probe if audio URL is valid

Make it possible to pass headers to _is_valid_url

* [infoq] Add audio only test
8 years ago
Sergey M․ e228616c6e
[extractor/common] Fix initialization template (closes #11605, closes #11825) 8 years ago
Sergey M․ c58c2d63cb
[extractor/common] Document forgotten fragment base and path interfaces 8 years ago
Sergey M․ d04621daf4
[extractor/common] Fix duration per dash segment (closes #11868) 8 years ago
Remita Amine c4251b9aaa [common] add possibility to customize akamai manifest host 8 years ago
Yen Chi Hsuan 4606c34e19 [extractor/common] Allow non-lang in subtitles' keys
See 264e77c406
8 years ago
Remita Amine a07588369f [common] improve detection for video only formats and m3u8 manifest(fixes #11507) 8 years ago
Remita Amine 7fe1592073 [common] fix dash codec information for mixed videos and fragment url construction(#11490) 8 years ago
Sergey M․ 87a449c1ed
[extractor/common] Recognize DASH formats in html5 media entries 8 years ago
Remita Amine 8821a718cf [common] recognize hls manifests that contain video only formats(#11394) 8 years ago
Sergey M․ 4ea4c0bb22
[extractor/common] Fix Bandwidth substitution in media template (closes #11175) 8 years ago
Sergey M․ f076d7972c
[extractor/common] Improve thumbnail extraction from JSON-LD 8 years ago
Sergey M․ c4c9b8440c
[extractor/common] Tolerate malformed RESOLUTION attribute in m3u8 manifests (closes #11113) 8 years ago
Sergey M․ 3365ea8929
[extractor/common] Remove unused code 8 years ago
Sergey M․ 1616f9b452
[extractor/common] Fix typo 8 years ago
Sergey M․ 02dc0a36b7
[utils] Introduce base_url 8 years ago
Remita Amine b2758123c5 add Basic support for Smooth Streaming protocol(#8118) 8 years ago
Sergey M․ b82c33dd67
[extractor/common] Improve mpd base URL extraction (closes #10909, closes #11079) 8 years ago
Remita Amine 0384932e3d [extractor/common] try to extract non smil wowza mpd manifests 8 years ago
Remita Amine edd6074cea [extractor/common] detect f4m audio only formats 8 years ago
Sergey M․ 8d76bdf12b
[extractor/common] Mention podcast in series fields section 8 years ago
Yen Chi Hsuan cea364f70c
[extractor/common] Support HTML media elements without child nodes 8 years ago
Yen Chi Hsuan 98763ee354
[extractor/common] Add id and title helpers for generic IEs 8 years ago
Remita Amine e71a450956 [common] add hdcore sign to akamai f4m formats 8 years ago
Yen Chi Hsuan 5968d7d2fe
[extractor/common] Improved support for HTML5 subtitles
Ref: #10625

In a strict sense, <track>s with kind=captions are not subtitles. [1]
openload misuses this attribute, and I guess there will be more
examples, so I add it to common.py.

Also allow extracting information for subtitles-only <video> or <audio>
tags, which is the case of openload.

[1] https://www.w3.org/TR/html5/embedded-content-0.html#attr-track-kind
8 years ago
Sergey M․ 30d0b549be
[extractor/common] Add manifest_url for hls and hds formats 8 years ago
Sergey M․ 86f4d14f81
Refactor fragments interface and dash segments downloader
- Eliminate segment_urls and initialization_url
+ Introduce manifest_url (manifest may contain unfragmented data in this case url will be used for direct media URL and manifest_url for manifest itself correspondingly)
* Rewrite dashsegments downloader to use fragments data
* Improve generic mpd extraction
8 years ago
Sergey M․ b4c1d6e800
[extractor/common] Expose fragments interface for dashsegments formats 8 years ago
Sergey M․ a0d5077c8d
[extractor/common] Introduce fragments interface 8 years ago
Remita Amine 6ad0219556 [common] add helper method for Wowza Streaming Engine format extraction 8 years ago
Sergey M․ dcce092e0a
[extractor/common] Simplify _get_netrc_login_info and carry long lines 8 years ago
Sergey M․ 32443dd346
[extractor/common] Update _get_login_info's comment 8 years ago
Sergey M․ 2133565cec
[extractor/common] Simplify _get_login_info 8 years ago
Sergey M․ c849836854
[utils] Improve _hidden_inputs 8 years ago
Remita Amine 1b6712ab23 [adobepass] add specific options for adobe pass authentication
- add --ap-username and --ap-password option to specify
TV provider username and password in the cmd line
- add --ap-retries option to limit the number of retries
- add --list-ap-msi-ids to list the supported TV Providers
8 years ago
Remita Amine 0def758782 [internetvideoarchive] extract all formats 8 years ago
Christian Pointner dedb177029
Fix parsing of HTML5 media elements
This fixes an error in _parse_html5_media_entries in case
an audio or video tag directly uses a src attribute insted
of <source> elements in it's body.
8 years ago
Sergey M․ e816c9d158
[extractor/common] Simplify _extract_m3u8_formats 8 years ago
Sergey M․ 9250181f37
[extractor/common] Restore NAME usage from EXT-X-MEDIA tag for formats codes in _extract_m3u8_formats (Closes #10522) 8 years ago
Remita Amine f39ffc5877 [common] extract formats from #EXT-X-MEDIA tags 8 years ago
Remita Amine c7c43a93ba [common] add helper method to extract akamai m3u8 and f4m formats 8 years ago
Yen Chi Hsuan ad120ae1c5
[extractor/common] Change the default m3u8 protocol in HTML5
Helper functions should have consistent default values
8 years ago
Yen Chi Hsuan 520251c093
[extractor/common] Recognize m3u8 manifests in HTML5 multimedia tags 8 years ago
Remita Amine 2118fdd1a9 [common] add separate method for getting netrc ligin info 8 years ago
Sergey M․ 321b5e082a
[extractor/common] Respect default in _search_json_ld 8 years ago
Remita Amine 37768f9242 [common] correctly lower the preference of m3u8 master manifest format 8 years ago
Remita Amine d16b3c6677 [common] extract partOfTVSeries info in json-ld 8 years ago
Remita Amine c57244cdb1 [common] lower the preference of m3u8 master manifest format 8 years ago
Sergey M․ 46933a15d6
[extractor/common] Support root JSON-LD lists (Closes #10203) 8 years ago
Remita Amine 8b40854529 [common] lower proto_preference of rtsp formats
Most of the time the RtspFD fail to download videos but it report
success of the download with this output:
[mpv] 0 bytes
[download] 100% of 0.00B
8 years ago
Sergey M․ b070564efb
[extractor/common] Support multiple properties in _og_search_property 8 years ago
Sergey M․ 40090e8d51
[extractor/common] Improve is_suitable
In order to fix breakage introduced by a3aa814b77
8 years ago
Sergey M․ f094834857
[extractor/common] Add support for $ in SegmentTemplate in MPD manifests 8 years ago
Remita Amine 00f4764cb7 [common] extract vbr, abr and fps for Unified Streaming Platform m3u8 manifests 8 years ago
Yen Chi Hsuan 0b68de3cc1 Merge pull request #8876 from remitamine/html5_media
[extractor/common] add helper method to extract html5 media entries
8 years ago
Sergey M․ 95b31e266b
[extractor/common] Add expected_type in json ld routines 8 years ago
Sergey M․ 6b3a3098b5
[extractor/common] Extract more metadata for VideoObject in _json_ld 8 years ago
Sergey M․ 2de624fdd5
[extractor/common] Introduce filesize metafield for thumbnails 8 years ago
Déstin Reed 3fee7f636c
[flipagram] Add extractor 8 years ago
Yen Chi Hsuan 38cce791c7
Rename --cn-verfication-proxy to --geo-verification-proxy
And deprecate the former one

Since commit f138873900, this option is
not limited to China websites, so rename it.
8 years ago
Sergey M․ bf8dd79045
[extractor/common] Fix sorting with custom field preference 9 years ago
remitamine 59bbe4911a [extractor/common] add helper method to extract html5 media entries 9 years ago
Sergey M․ 88d9f6c0c4
[utils] Add support for name list in _html_search_meta 9 years ago
remitamine e154c65128 [downloader/hls] Add support for AES-128 encrypted segments in hlsnative downloader 9 years ago
Yen Chi Hsuan a6571f1073
[common] Fix <bootstrapInfo> detection in F4M manifests
Regression since 0a5685b26f
9 years ago
Sergey M․ 77b8b4e696
[extractor/common] Borrow quality metadata from parent set-level manifest for f4m 9 years ago
Yen Chi Hsuan fac2af3c51
[common] Fix m3u8 extraction in f4m manifests 9 years ago
Yen Chi Hsuan 448bb5f333
[common] Fix non-bootstrapped support in f4m 9 years ago
Yen Chi Hsuan 240b60453e
[common] Support m3u8 in f4m manifests
Related: #9531
9 years ago
Yen Chi Hsuan 0a5685b26f
[common] Support non-bootstraped streams in f4m manifests
Related: #9531
9 years ago
Yen Chi Hsuan 7b2fcbfd4e
[common] Skip TYPE=CLOSED-CAPTIONS lines in m3u8 manifests
According to [1], valid values for TYPE are AUDIO, VIDEO, SUBTITLES
and CLOSED-CAPTIONS. Such a value is found in Anvato master playlists,
though I don't use _extract_m3u8_formats() in the end.

Part of #9522.

[1] https://tools.ietf.org/html/draft-pantos-http-live-streaming-19#section-4.3.4.1
9 years ago
Yen Chi Hsuan 16da9bbc29
[common] Add _m3u8_meta_format() template
For extractors who handle m3u8 manifests by themselves. (eg., AnvatoIE)

Part of #9522
9 years ago
Yen Chi Hsuan ad96b4c8f5
[common] Extract audio formats in SMIL
Found in http://www.cbc.ca/player/play/2657631896

Closes #5156
9 years ago
Sergey M․ ed56f26039
[extractor/common] Improve name extraction for m3u8 formats 9 years ago
Sergey M․ 8a92e51c60
[extractor/common] Relax wording for creator metafield 9 years ago
Yen Chi Hsuan e9c6cdf4a1
[common] Fix format_id construction for HLS 9 years ago
Kagami Hiiragi b24d6336a7 [vlive] Add support for live videos 9 years ago
Yen Chi Hsuan d6712378e7
Merge branch 'akamai_pv' of https://github.com/remitamine/youtube-dl into remitamine-akamai_pv 9 years ago
remitamine fb72ec58ae [extractor/common] do not process f4m manifest that contain akamai playerVerificationChallenge 9 years ago
Yen Chi Hsuan 2c0d9c6217
[extractor/common] Allow empty post data 9 years ago
Sergey M․ 49caf3307f
[extractor/common] Remove irrelevant comment 9 years ago
Sergey M․ bacec0397f [extractor/common] Relax _hidden_inputs 9 years ago
Sergey M․ fb38aa8b53 [extractor/common] Support arbitrary format strings for template based identifiers in mpd manifests (Closes #9119, closes #9120) 9 years ago
Sergey M․ 7a93ab5f3f [extractor/common] Introduce music album metafields 9 years ago
Sergey M․ b507cc925b [extractor/common] Carry long line 9 years ago
Sergey M․ db8ee7ec05 [extractor/common] Fix numeric identifiers conversion in DASH URL templates 9 years ago
remitamine df634be2ed [common] prefer using mime type over ext for smil subtitle extraction
the subtitle ext for http://www.cnet.com/videos/download-amazon-prime-movies-and-tv/
is adb_xml while using the mime type it get tt(application/smptett+xml)
9 years ago
Sergey M․ 41d06b0424 [extractor/common] Improve _request_webpage
* Do not ignore data, headers and query for Requests
* Default values for headers and query switched to dicts since these are used by urllib itself
9 years ago
Sergey M․ b22ca76204 [extractor/common] Filter out unsupported encrypted media for f4m formats (Closes #8573) 9 years ago
Sergey M․ 19dbaeece3 Remove _sort_formats from _extract_*_formats methods
Now _sort_formats should be called explicitly.
_sort_formats has been added to all the necessary places in code.

Closes #8051
9 years ago
Sergey M․ 15707c7e02 [compat] Add compat_urllib_parse_urlencode and eliminate encode_dict
encode_dict functionality has been improved and moved directly into compat_urllib_parse_urlencode
All occurrences of compat_urllib_parse.urlencode throughout the codebase have been replaced by compat_urllib_parse_urlencode

Closes #8974
9 years ago
remitamine 49dea4913b Merge pull request #8513 from remitamine/dash-sort
[extractor/common] fix dash formats sorting
9 years ago
Sergey M․ 0fdbb3322b [extractor/common] Add _parse_f4m_formats routine 9 years ago
remitamine 09f572fbc0 [extractor/common] add transform_source to _download_smil and _extract_smil_formats 9 years ago
remitamine 15bf934de5 Merge pull request #8819 from remitamine/simple-webpage-requests
[extractor/common] simplify using data, headers and query params with _download_* methods
9 years ago
remitamine cdfee16818 [extractor/common] add data, headers and query params to _request_webpage 9 years ago
Yen Chi Hsuan a6c8b75904 [common] Use mimeType to determine file extensions (#8766) 9 years ago
Yen Chi Hsuan 64f08d4ff2 Merge pull request #8766 from yan12125/dash-detect-ext
Detect file extensions of DASH formats from their codecs
9 years ago
Yen Chi Hsuan af7d5a63b2 [common] Document protocol http_dash_segments 9 years ago
Yen Chi Hsuan 2def60c5f3 [common] Use codec2ext for DASH formats (#8764) 9 years ago
Yen Chi Hsuan e9c0cdd389 [jython] Introduce compat_os_name
os.name is always 'java' on Jython
9 years ago
Sergey M․ 7bcd2830dd [extractor/common] Document uploader_url 9 years ago
Sergey M․ 2bc0c46f98 [extractor/common] Document license metafield 9 years ago
Sergey M․ d77ab8e255 Add --mark-watched feature (Closes #5054) 9 years ago
Sergey M․ 9cdffeeb3f [extractor/common] Clarify rationale on media playlist detection 9 years ago
Sergey M․ fbb6edd298 [extractor/common] Properly extract audio only formats in master m3u8 playlists 9 years ago
Sergey M․ f5bdb44443 [extractor/common] Add _remove_duplicate_formats 9 years ago
remitamine cafcf657a4 add more subtitles mime types to mimetype2ext and fix the platform subtitle extraction 9 years ago
Sergey M․ 611c1dd96e [refactor] Single quotes consistency 9 years ago
Sergey M․ d800609c62 [refactor] Do not specify redundant None as second argument in dict.get() 9 years ago
Sergey M․ bb20526b64 [extractor/common] Improve base url construction 9 years ago
remitamine c349456ef6 [extractor/common] strip http urls in smil manifest 9 years ago
remitamine 81e1c4e2fc [extractor/common] remove duplicate rtmp formats in smil manifest 9 years ago
remitamine dd86780596 [extractor/common] fix dash formats sorting 9 years ago
remitamine 154c209e2d [extractor/common] improve dash format ids 9 years ago
remitamine 51e9094f4a [extractor/common] extract youtube dash formats filesize(fixes #8480) 9 years ago
remitamine d413095f7e [extractor/common] remove duplicated formats and subtiles in smil manifests 9 years ago
remitamine 6a3828fddd [common] use float conversion instead of using division from __future__ 9 years ago
remitamine 91cb6b5065 rename _parse_mpd to _parse_mpd_formats and add default value for mpd namespace 9 years ago
remitamine 0826a0b555 [common] sort dash formats 9 years ago
remitamine 255732f0d3 [common] fix segment duration calculation 9 years ago
remitamine 53c269c6fd [common] fix media_template string formating 9 years ago
remitamine 675d001633 [common] skip drm protected dash formats 9 years ago
remitamine d577c79632 [common] ignore ISO 639-2 generic codes 9 years ago
remitamine f14be22816 [common] remove duplicate reference to namespace 9 years ago
remitamine 9c74423510 [common] fix media template regex 9 years ago
remitamine 1bac34556f [common] add a generic support for mpd manifests 9 years ago
Yen Chi Hsuan 2d2fa82d17 [common] Add _extract_dash_manifest_formats 9 years ago
Yen Chi Hsuan c94678957f [common] Remove unused arguments 9 years ago
Yen Chi Hsuan 16f38a699f [common] Rename to namespace
For consistency with _parse_smil_*
9 years ago