henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
766 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'f8d88645a3'
${ noResults }
Commit Graph

766 Commits (f8d88645a30eac2ecc7319f719603b616670a0b6)
 

Author SHA1 Message Date
Mike Lang cb75953e91 thrimshim: Fix a bug preventing submissions 
When comparing old and new video tags, it errors because it's a list, not string.

We change it to apply the transforms to all tags in the list, and also ignore changes in list ordering.
 4 years ago
Mike Lang 9d8c47377f segment parsing: Hand-roll microsecond parsing 
float() is inaccurate and Decimal() is very slow (~3x the cpu usage)
so instead we right-pad with 0s (eg. so 1.2345 -> 1.234500) then convert to int microsec directly.
 4 years ago
Christopher Usher 41d85b8614 Remove unicode 4 years ago
Christopher Usher 84f7ef71c1 More helpful UI if the database has changed plus ignore changes that are 
just white space or capitalisation
 4 years ago
Mike Lang 31d241eecc thrimbletrimmer: missing tags arg when submitting 4 years ago
Mike Lang 66669cd4e4 common: When parsing segment timestamps, use decimal instead of float 
Floating point error leads to 1us differences in parsed times,
which causes false positives in the overlapping segments check.

By using a Decimal, we get the exact digits from the filepath.
 4 years ago
Mike Lang 389d3e08d7 Fix typo in making playlist manager case insensitive 4 years ago
Mike Lang 3a19ba744d segment_coverage: Raise default check interval to 5min 4 years ago
Mike Lang 5235c3281a segment_coverage: Allow setting of check interval via cli flag 4 years ago
Mike Lang 15c357509f segment_coverage: Fix a problem where metrics would fail 
Because the checking process is entirely CPU-bound, it does not give any other
greenlets a chance to run while it is processing. This prevents us from responding
to metrics queries, and prometheus then times out.

By stopping to handle all other traffic in between each hour processed, we ensure metrics
remain responsive while processing.
 4 years ago
Mike Lang 13a228070a common.segments: Speed up segment parsing by rolling our own time parsing 
strptime is very slow. In terms of pure get_best_segments() speed, this change
more than doubles the throughput.

In particular for segment_coverage, this halves the run time for each check.
 4 years ago
Mike Lang e9b2831b71 docker-compose: Fix bug preventing playlist manager metrics from being routed 4 years ago
Mike Lang efe185bc0c Fix copy-paste error in restreamer metrics endpoint 4 years ago
Mike Lang fa1f305fda postgres: Changes to work with version 12 
In postgres 12, recovery.conf is removed in favor of normal config options
plus a signal file.

See https://www.2ndquadrant.com/en/blog/replication-configuration-changes-in-postgresql-12/
for a good rundown.
 4 years ago
Mike Lang a044b30465 postgres: Fix replica not working when there are spaces in the password 4 years ago
Mike Lang ab33dfe00d Fix some typos in playlist manager 4 years ago
Mike Lang 2327c7c9a8 docker-compose: Fix typos in configuring playlist manager 4 years ago
Mike Lang 5357a40ea7 playlist_manager: Compare tags case-insensitively 
To avoid confusion between eg. "Interview" and "interview"
 4 years ago
HubbeKing 421a9ab42e Actually fix serviceName for segment_coverage ingress 
Turns out I can't type. At all.
 4 years ago
HubbeKing f7f5523d5d Fix serviceName for segment_coverage ingress rule 4 years ago
Mike Lang 48ef416dfb restreamer, thrimshim: Allow /metrics/* in addition to /metrics 
This allows the metrics proxying from nginx to work even if the path is not rewritten,
which is hard to do in k8s.
 4 years ago
Mike Lang fbcfea745a k8s.jsonnet: Fix metrics rules 
restreamer should have a metrics rule, and segment_coverage should be with an underscore.
 4 years ago
Mike Lang 6e5e4b41df docker-compose/k8s: Limit segment-coverage start/end times 
Instead of defaulting to oldest/newest.
 4 years ago
Christopher Usher c19dce6e4e I forgot to add tags 4 years ago
Christopher Usher 27c6b70976 fixes in response to review 4 years ago
Christopher Usher b21a520d3a Checking for changes in the sheet now working in the thrimbletrimmer 4 years ago
Christopher Usher 1216c8e8a5 thrimshim side of checking for changes seems to be working 4 years ago
Christopher Usher 8889648a58 Check whether sheet columns have changed during editing in thrimshim. 4 years ago
Mike Lang 7981623611 k8s.jsonnet: Default to only backfilling things from this year 
By setting a 6-month limit
 4 years ago
Mike Lang 324d180c5e docker-compose: Default to only backfilling things from this year 
By setting a 6-month limit
 4 years ago
Mike Lang b029250c1c Disable stacksampler by default 
It causes problems due to the sheer number of unique metrics emitted, which makes
the prometheus endpoint be very expensive / fail a lot.

The data is not useful enough to justify the cost.
 4 years ago
Mike Lang 6d55f01de6 downloader: Fix a bug when we can't find a particular stream quality 
The intended behaviour was to log a warning message and retry next time,
but still allow workers to be started for any streams found.

However, due to a missing continue, we fall through to attempting to start a worker
for a non-existent quality which causes a KeyError when looking up
`self.latest_urls[quality]`. This exception means we don't run through the other qualities,
so we never start any other quality.
 4 years ago
Hubbe d2713103a9
Add thrimshim to k8s.jsonnet (#185) 
* Add thrimshim to k8s jsonnet file

* Fix reference to bustime_start in thrimshim

* Add "enabled" config to selectively disable things

* Fix styling and handling of disabled components

* Don't need to "hide" enabled field

* Add port arg to thrimshim deployment

* fix indent nitpick

Co-authored-by: Mike Lang <ekimekim@users.noreply.github.com>
 4 years ago
Mike Lang b53fcd65a0 Add dependencies required to install psycopg2 from source 
We can't install the binaries as they don't support musl
 4 years ago
Mike Lang a89bfcd271 Add playlist manager to all the various other places 4 years ago
Mike Lang 9413b99b91 Implement playlist manager 4 years ago
Mike Lang 9562032c24 Create new playlist manager service 
Playlist manager adds youtube videos to youtube playlists based on database tags.
 4 years ago
HubbeKing 86f7823348 Replace calls to gevent.signal() with gevent.signal_handler() 
gevent.signal() was removed in gevent 1.5a4, see http://www.gevent.org/api/gevent.signal.html
Removed on Feb 5th, see https://github.com/gevent/gevent/pull/1530
 4 years ago
Mike Lang a53786dc2d Add file and make as build dependencies 
gevent now requires these to build. I'm not sure when this changed.
 4 years ago
Mike Lang 5ccb5afff1 Track number of ignored ads 4 years ago
Mike Lang 50770651ce Detect new style of twitch in-stream ads 
New ad segments always have a title like "Amazon|...".
This is the same fix as used by streamlink at time of writing.
 4 years ago
Mike Lang b9cd76b1a2 Add non-static implict tags in sheetsync 
In order for the upcoming playlist manager to be able to use the DB `tags` column to know
what tags a video has, all the tags it needs need to be present.

Previously, this was a problem because the day and category tags only get added at the cutter
and so wouldn't be listed.

This moves them so they are added when parsing the row in sheetsync.
It also adds the poster moment tag if poster moment is checked.

Note that fully static tags that go on all videos are still only added in cutter,
but the playlist manager doesn't need to care about those (since by definition
they will match every video).
 4 years ago
Mike Lang a30d595fee cutter: Use video's custom tags when uploading 4 years ago
Mike Lang c26a87565f Add video tags to thrimbletrimmer 
Tags default to tags given on the sheet, but can be modified by the editor.
Tags are represented as a comma-seperated string, and are round-tripped on loss of focus
as a way to validate.
 4 years ago
Mike Lang 7e0c415f83 thrimshim: Add video tags column as required 4 years ago
Mike Lang 29571fb60b Add tags column to sheetsync 
New tags column shunts all columns after it right by 1.

Note we parse tags by splitting on commas then discarding whitespace.
If this would create an empty string tag, it is ignored.
Example: "foo, bar baz,a,,bc " -> ["foo", "bar baz", "a", "bc"]
 4 years ago
Mike Lang 68b75fa891 Add tags and video_tags database columns 
tags is a sheet input which provides a default list of tags for the editor.
video_tags is set upon the video being edited and is used by the cutter to set video tags on supported upload locations.
 4 years ago
Hubbe 521b193584
Merge pull request #178 from ekimekim/k8s-ingress-tls 
Add TLS settings to k8s.jsonnet
 4 years ago
HubbeKing 9eca34c7a9 Fix suggested style changes 4 years ago
HubbeKing 4ca425dd12 Simplify ingress_secret_name setting and add ingress_labels setting 
using std.objectHas() removes the need to have a separate boolean to check.
 4 years ago