henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
974 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '546572a697'
${ noResults }
Commit Graph

974 Commits (546572a6975b95a3f2ddbc607cdb39e41c3b4708)
 

Author SHA1 Message Date
Mike Lang 32138bbd43 downloader: Update to work with twitch's new access token API 
Twitch removed their old access token endpoint and now use a GraphQL endpoint.
The old endpoint would just always return 404, which we sadly interpreted as "stream not up".

Thankfully streamlink has already done the reverse engineering work so I was able to
update it to work again fairly easily, it's just a bit more convoluted.
 4 years ago
Hubbe 36a5387e26 Change wording on edit submitted alert() 
An edit has been submitted for the video. The video hasn't been submitted yet, thrimbletrimmer just informs other components how it wants the edit to be.
 4 years ago
Mike Lang 74a38bfaeb thrimbletrimmer: Instead of redirecting on submit, alert but stay 4 years ago
Mike Lang 337111a9ab thrimbletrimmer: Always alert() on errors 4 years ago
Hubbe 27265f3bca Set video.js volume to 50% by default 
When the video.js player is ready, set its volume to 50%
 4 years ago
Mike Lang f8c877775d Add an alternate manual upload mode specifically for youtube 
Adds a built-in "youtube-manual" location which is like "manual" except that it only works
with youtube URLs and populates the video_id column.

The intent is so that we can have playlist_manager manage videos we upload manually,
while still being able to distinguish between that and other manual links that shouldn't
be included (eg. links to third party youtube videos).

This is set when setting a manual link in thrimbletrimmer with a new checkbox, default off.
 4 years ago
Christopher Usher ad03a087ab Added tags to the thrimbletrimmer dashboard 4 years ago
Mike Lang 6d789ad42f thrimshim: Handle None values in comparing columns 4 years ago
Mike Lang cb75953e91 thrimshim: Fix a bug preventing submissions 
When comparing old and new video tags, it errors because it's a list, not string.

We change it to apply the transforms to all tags in the list, and also ignore changes in list ordering.
 4 years ago
Mike Lang 9d8c47377f segment parsing: Hand-roll microsecond parsing 
float() is inaccurate and Decimal() is very slow (~3x the cpu usage)
so instead we right-pad with 0s (eg. so 1.2345 -> 1.234500) then convert to int microsec directly.
 4 years ago
Christopher Usher 41d85b8614 Remove unicode 4 years ago
Christopher Usher 84f7ef71c1 More helpful UI if the database has changed plus ignore changes that are 
just white space or capitalisation
 4 years ago
Mike Lang 31d241eecc thrimbletrimmer: missing tags arg when submitting 4 years ago
Mike Lang 66669cd4e4 common: When parsing segment timestamps, use decimal instead of float 
Floating point error leads to 1us differences in parsed times,
which causes false positives in the overlapping segments check.

By using a Decimal, we get the exact digits from the filepath.
 4 years ago
Mike Lang 389d3e08d7 Fix typo in making playlist manager case insensitive 4 years ago
Mike Lang 3a19ba744d segment_coverage: Raise default check interval to 5min 4 years ago
Mike Lang 5235c3281a segment_coverage: Allow setting of check interval via cli flag 4 years ago
Mike Lang 15c357509f segment_coverage: Fix a problem where metrics would fail 
Because the checking process is entirely CPU-bound, it does not give any other
greenlets a chance to run while it is processing. This prevents us from responding
to metrics queries, and prometheus then times out.

By stopping to handle all other traffic in between each hour processed, we ensure metrics
remain responsive while processing.
 4 years ago
Mike Lang 13a228070a common.segments: Speed up segment parsing by rolling our own time parsing 
strptime is very slow. In terms of pure get_best_segments() speed, this change
more than doubles the throughput.

In particular for segment_coverage, this halves the run time for each check.
 4 years ago
Mike Lang e9b2831b71 docker-compose: Fix bug preventing playlist manager metrics from being routed 4 years ago
Mike Lang efe185bc0c Fix copy-paste error in restreamer metrics endpoint 4 years ago
Mike Lang fa1f305fda postgres: Changes to work with version 12 
In postgres 12, recovery.conf is removed in favor of normal config options
plus a signal file.

See https://www.2ndquadrant.com/en/blog/replication-configuration-changes-in-postgresql-12/
for a good rundown.
 4 years ago
Mike Lang a044b30465 postgres: Fix replica not working when there are spaces in the password 4 years ago
Mike Lang ab33dfe00d Fix some typos in playlist manager 4 years ago
Mike Lang 2327c7c9a8 docker-compose: Fix typos in configuring playlist manager 4 years ago
Mike Lang 5357a40ea7 playlist_manager: Compare tags case-insensitively 
To avoid confusion between eg. "Interview" and "interview"
 4 years ago
HubbeKing 421a9ab42e Actually fix serviceName for segment_coverage ingress 
Turns out I can't type. At all.
 4 years ago
HubbeKing f7f5523d5d Fix serviceName for segment_coverage ingress rule 4 years ago
Mike Lang 48ef416dfb restreamer, thrimshim: Allow /metrics/* in addition to /metrics 
This allows the metrics proxying from nginx to work even if the path is not rewritten,
which is hard to do in k8s.
 4 years ago
Mike Lang fbcfea745a k8s.jsonnet: Fix metrics rules 
restreamer should have a metrics rule, and segment_coverage should be with an underscore.
 4 years ago
Mike Lang 6e5e4b41df docker-compose/k8s: Limit segment-coverage start/end times 
Instead of defaulting to oldest/newest.
 4 years ago
Christopher Usher c19dce6e4e I forgot to add tags 4 years ago
Christopher Usher 27c6b70976 fixes in response to review 4 years ago
Christopher Usher b21a520d3a Checking for changes in the sheet now working in the thrimbletrimmer 4 years ago
Christopher Usher 1216c8e8a5 thrimshim side of checking for changes seems to be working 4 years ago
Christopher Usher 8889648a58 Check whether sheet columns have changed during editing in thrimshim. 4 years ago
Mike Lang 7981623611 k8s.jsonnet: Default to only backfilling things from this year 
By setting a 6-month limit
 4 years ago
Mike Lang 324d180c5e docker-compose: Default to only backfilling things from this year 
By setting a 6-month limit
 4 years ago
Mike Lang b029250c1c Disable stacksampler by default 
It causes problems due to the sheer number of unique metrics emitted, which makes
the prometheus endpoint be very expensive / fail a lot.

The data is not useful enough to justify the cost.
 4 years ago
Mike Lang 6d55f01de6 downloader: Fix a bug when we can't find a particular stream quality 
The intended behaviour was to log a warning message and retry next time,
but still allow workers to be started for any streams found.

However, due to a missing continue, we fall through to attempting to start a worker
for a non-existent quality which causes a KeyError when looking up
`self.latest_urls[quality]`. This exception means we don't run through the other qualities,
so we never start any other quality.
 4 years ago
Hubbe d2713103a9
Add thrimshim to k8s.jsonnet (#185) 
* Add thrimshim to k8s jsonnet file

* Fix reference to bustime_start in thrimshim

* Add "enabled" config to selectively disable things

* Fix styling and handling of disabled components

* Don't need to "hide" enabled field

* Add port arg to thrimshim deployment

* fix indent nitpick

Co-authored-by: Mike Lang <ekimekim@users.noreply.github.com>
 4 years ago
Mike Lang b53fcd65a0 Add dependencies required to install psycopg2 from source 
We can't install the binaries as they don't support musl
 4 years ago
Mike Lang a89bfcd271 Add playlist manager to all the various other places 4 years ago
Mike Lang 9413b99b91 Implement playlist manager 4 years ago
Mike Lang 9562032c24 Create new playlist manager service 
Playlist manager adds youtube videos to youtube playlists based on database tags.
 4 years ago
HubbeKing 86f7823348 Replace calls to gevent.signal() with gevent.signal_handler() 
gevent.signal() was removed in gevent 1.5a4, see http://www.gevent.org/api/gevent.signal.html
Removed on Feb 5th, see https://github.com/gevent/gevent/pull/1530
 4 years ago
Mike Lang a53786dc2d Add file and make as build dependencies 
gevent now requires these to build. I'm not sure when this changed.
 4 years ago
Mike Lang 5ccb5afff1 Track number of ignored ads 4 years ago
Mike Lang 50770651ce Detect new style of twitch in-stream ads 
New ad segments always have a title like "Amazon|...".
This is the same fix as used by streamlink at time of writing.
 4 years ago
Mike Lang b9cd76b1a2 Add non-static implict tags in sheetsync 
In order for the upcoming playlist manager to be able to use the DB `tags` column to know
what tags a video has, all the tags it needs need to be present.

Previously, this was a problem because the day and category tags only get added at the cutter
and so wouldn't be listed.

This moves them so they are added when parsing the row in sheetsync.
It also adds the poster moment tag if poster moment is checked.

Note that fully static tags that go on all videos are still only added in cutter,
but the playlist manager doesn't need to care about those (since by definition
they will match every video).
 4 years ago