henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
696 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '714a16bac4'
${ noResults }
Commit Graph

696 Commits (714a16bac4c56cc477a532d85a7f1d6efb7d2196)
 

Author SHA1 Message Date
Mike Lang 48ef416dfb restreamer, thrimshim: Allow /metrics/* in addition to /metrics 
This allows the metrics proxying from nginx to work even if the path is not rewritten,
which is hard to do in k8s.
 4 years ago
Mike Lang fbcfea745a k8s.jsonnet: Fix metrics rules 
restreamer should have a metrics rule, and segment_coverage should be with an underscore.
 4 years ago
Mike Lang 6e5e4b41df docker-compose/k8s: Limit segment-coverage start/end times 
Instead of defaulting to oldest/newest.
 4 years ago
Christopher Usher c19dce6e4e I forgot to add tags 4 years ago
Christopher Usher 27c6b70976 fixes in response to review 4 years ago
Christopher Usher b21a520d3a Checking for changes in the sheet now working in the thrimbletrimmer 4 years ago
Christopher Usher 1216c8e8a5 thrimshim side of checking for changes seems to be working 4 years ago
Christopher Usher 8889648a58 Check whether sheet columns have changed during editing in thrimshim. 4 years ago
Mike Lang 7981623611 k8s.jsonnet: Default to only backfilling things from this year 
By setting a 6-month limit
 4 years ago
Mike Lang 324d180c5e docker-compose: Default to only backfilling things from this year 
By setting a 6-month limit
 4 years ago
Mike Lang b029250c1c Disable stacksampler by default 
It causes problems due to the sheer number of unique metrics emitted, which makes
the prometheus endpoint be very expensive / fail a lot.

The data is not useful enough to justify the cost.
 4 years ago
Mike Lang 6d55f01de6 downloader: Fix a bug when we can't find a particular stream quality 
The intended behaviour was to log a warning message and retry next time,
but still allow workers to be started for any streams found.

However, due to a missing continue, we fall through to attempting to start a worker
for a non-existent quality which causes a KeyError when looking up
`self.latest_urls[quality]`. This exception means we don't run through the other qualities,
so we never start any other quality.
 4 years ago
Hubbe d2713103a9
Add thrimshim to k8s.jsonnet (#185) 
* Add thrimshim to k8s jsonnet file

* Fix reference to bustime_start in thrimshim

* Add "enabled" config to selectively disable things

* Fix styling and handling of disabled components

* Don't need to "hide" enabled field

* Add port arg to thrimshim deployment

* fix indent nitpick

Co-authored-by: Mike Lang <ekimekim@users.noreply.github.com>
 4 years ago
Mike Lang b53fcd65a0 Add dependencies required to install psycopg2 from source 
We can't install the binaries as they don't support musl
 4 years ago
Mike Lang a89bfcd271 Add playlist manager to all the various other places 4 years ago
Mike Lang 9413b99b91 Implement playlist manager 4 years ago
Mike Lang 9562032c24 Create new playlist manager service 
Playlist manager adds youtube videos to youtube playlists based on database tags.
 4 years ago
HubbeKing 86f7823348 Replace calls to gevent.signal() with gevent.signal_handler() 
gevent.signal() was removed in gevent 1.5a4, see http://www.gevent.org/api/gevent.signal.html
Removed on Feb 5th, see https://github.com/gevent/gevent/pull/1530
 4 years ago
Mike Lang a53786dc2d Add file and make as build dependencies 
gevent now requires these to build. I'm not sure when this changed.
 4 years ago
Mike Lang 5ccb5afff1 Track number of ignored ads 4 years ago
Mike Lang 50770651ce Detect new style of twitch in-stream ads 
New ad segments always have a title like "Amazon|...".
This is the same fix as used by streamlink at time of writing.
 4 years ago
Mike Lang b9cd76b1a2 Add non-static implict tags in sheetsync 
In order for the upcoming playlist manager to be able to use the DB `tags` column to know
what tags a video has, all the tags it needs need to be present.

Previously, this was a problem because the day and category tags only get added at the cutter
and so wouldn't be listed.

This moves them so they are added when parsing the row in sheetsync.
It also adds the poster moment tag if poster moment is checked.

Note that fully static tags that go on all videos are still only added in cutter,
but the playlist manager doesn't need to care about those (since by definition
they will match every video).
 4 years ago
Mike Lang a30d595fee cutter: Use video's custom tags when uploading 4 years ago
Mike Lang c26a87565f Add video tags to thrimbletrimmer 
Tags default to tags given on the sheet, but can be modified by the editor.
Tags are represented as a comma-seperated string, and are round-tripped on loss of focus
as a way to validate.
 4 years ago
Mike Lang 7e0c415f83 thrimshim: Add video tags column as required 4 years ago
Mike Lang 29571fb60b Add tags column to sheetsync 
New tags column shunts all columns after it right by 1.

Note we parse tags by splitting on commas then discarding whitespace.
If this would create an empty string tag, it is ignored.
Example: "foo, bar baz,a,,bc " -> ["foo", "bar baz", "a", "bc"]
 4 years ago
Mike Lang 68b75fa891 Add tags and video_tags database columns 
tags is a sheet input which provides a default list of tags for the editor.
video_tags is set upon the video being edited and is used by the cutter to set video tags on supported upload locations.
 4 years ago
Hubbe 521b193584
Merge pull request #178 from ekimekim/k8s-ingress-tls 
Add TLS settings to k8s.jsonnet
 4 years ago
HubbeKing 9eca34c7a9 Fix suggested style changes 4 years ago
HubbeKing 4ca425dd12 Simplify ingress_secret_name setting and add ingress_labels setting 
using std.objectHas() removes the need to have a separate boolean to check.
 4 years ago
HubbeKing cf7670d008 More sensible default secretName for ingress TLS 
The given secretName has to be in the same namespace as the ingress itself.
 4 years ago
HubbeKing 1dc62c193f Add TLS settings to k8s.jsonnet 
Use jsonnet computed field names to optionally add TLS configuration to generated Ingress object

In this way, one can easily let the kubernetes ingress handle TLS, with or without a secretName

Additional configuration would be required to tie into cert-manager for automated cert generation
 4 years ago
Mike Lang d21b225a70
Merge pull request #177 from ekimekim/mike/fix-thrim-bug 
Fix a bug when we fail to load a playlist that prevents loading other playlists
 4 years ago
Mike Lang a37d9a0c54 Fix a bug when we fail to load a playlist that prevents loading other playlists 4 years ago
Mike Lang 46d96bf900
Merge pull request #173 from ekimekim/k8s 
Add alternate deploy stack for kubernetes
 4 years ago
Mike Lang 239fc5d9ad
Merge pull request #176 from ekimekim/mike/update-clock 
Update clock.html for 2020 and display negative time properly
 4 years ago
Mike Lang d2d457a45f Add k8s option to README 4 years ago
Mike Lang b248408842 Update clock.html for 2020 and display negative time properly 4 years ago
Mike Lang d6dbf5512d
Merge pull request #175 from ekimekim/sheet-updates 
sheetsync: Move column indexes to match updated sheet
 4 years ago
Mike Lang 7600ce67a9
Merge pull request #174 from ekimekim/mike/generate-videos 
Add restreamer call to generate full concat'd videos
 4 years ago
Mike Lang b85296a81e sheetsync: Move column indexes to match updated sheet 
New tags column shunts all columns after it right by 1.
We will later want to parse that, but for now we ignore it.
 4 years ago
Mike Lang 61d934f333 docker-compose.jsonnet: Use seperate tag for database 
The database changes rarely, and is disruptive to re-deploy,
so we track its image version seperately so it only needs to be upgraded
when there's actually a change.
 4 years ago
Mike Lang 1b12c05e0e make smart cut work, only to discover it doesn't actually have any advantage over fast 4 years ago
Mike Lang b47eb3865f Fix segment_coverage -> segment-coverage 
because _ is illegal in k8s names.

However the image name does contain a _ so we replace - with _ when constructing the image name.
 4 years ago
HubbeKing 9a95dc44eb Fix k8s.jsonnet to compile to something kubernetes accepts 
Add Service definitions to output
Correct apiVersion for Ingress definition
Output a v1 List, as kubectl doesn't parse JSON arrays
 4 years ago
Mike Lang 28faff5c58 Add a k8s version of the docker-compose file 
Note this version is very simplified compared to the docker-compose
and has some major limitations:
* It relies on hostPath and a nodeSelector to put all the components on a shared storage node
* It only supports use as a replication node (downloader, restreamer, backfiller, segment_coverage)
* It uses the k8s Ingress instead of the built-in nginx for http routing.
 4 years ago
Mike Lang 4d21f447ad Add restreamer call to generate full concat'd videos 
This makes the timelapse easier.
 5 years ago
Mike Lang dea143cc0a
Merge pull request #163 from ekimekim/mike/api-ping 
api_ping: A simple util for hitting google apis to prevent inactivity
 5 years ago
Mike Lang 3489b0abaf api_ping: A simple util for hitting google apis to prevent inactivity 5 years ago
Mike Lang 2dbd1132fe common.googleapis: Fix a bug in retrying failed access token get 
Seems that this was never fixed when the code was moved.
 5 years ago