henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
871 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'd8a9b5ddf0'
${ noResults }
Commit Graph

871 Commits (d8a9b5ddf0d8ed77a62f5dbbe8ff5785375915e0)
 

Author SHA1 Message Date
Mike Lang 8ae600f2e5 nginx: Prevent nginx from caching dns results 
Otherwise if the containers get restarted and change ip, nginx hits the wrong ip.

We do this via a hack where we make all references indirect through a variable.
Since nginx only resolves this at request time, it always does a dns request.
 5 years ago
Mike Lang 5ed8cf0a33
Merge pull request #147 from ekimekim/chrusher-downloader-label-fix 
fixed outdated "stream", "variant" in metric
 5 years ago
Mike Lang 7b6f691d63
Merge pull request #146 from ekimekim/mike/unicode/hack 
Automatically set default encoding to utf-8 when common is imported
 5 years ago
Christopher Usher abb9193705 fixed outdated "stream", "variant" in metric 5 years ago
Mike Lang 4d3aa94a71 Automatically set default encoding to utf-8 when common is imported 
To be clear, this is an awful hack.

It means that any implicit str/unicode coersion will use the utf-8 encoding,
which is basically always what you want.

However, it is possible that some badly-written libraries might be relying
on the default encoding being ascii, and will do weird things as a result.

Finally, it's especially hacky to be doing this as part of importing a library.
Normally you're meant to do this as part of a sitecustomize.py in your python system directory,
and the function is deleted before passing control to normal code (this is why we need
to reload() to get it back).
 5 years ago
Mike Lang f452aa3c32
Merge pull request #143 from ekimekim/mike/sheetsync/api-usage 
sheetsync improvements
 5 years ago
Mike Lang 1c0f3a627b sheetsync: Log what worksheets got synced 
it's kinda important
 5 years ago
Mike Lang 8b25f8be95 sheetsync: Inject an error into the error column if we fail to parse an input column 5 years ago
Mike Lang 8dc7b80de9 sheetsync: Improve timing of main loop 
Instead of always waiting 5 seconds between runs,
wait until 5 seconds after the previous run started.

This ensures we actually run every 5sec and not every 5sec + how long it takes to run
 5 years ago
Mike Lang cda8078f64 sheetsync: Only check the most recently changed two sheets most times 
Only check the other sheets every 4th time (20sec instead of 5sec).

This elminiates a huge source of unnessecary reads, which prevents us from going over
our API limit.
 5 years ago
Christopher Usher dfe9d61428 Update database install instructions 
Turns out postgres ignores the column order when reading from CSV so
update documenation to give the correct order.
 5 years ago
Mike Lang 48d4dddb1c
Merge pull request #142 from ekimekim/mike/fixes 
Misc fixes and improvements
 5 years ago
Mike Lang 07b93f779c replace readlink -f with realpath in build scripts 
readlink -f doesn't work on OS X
 5 years ago
Mike Lang 520f7821a6 thrimbletrimmer: Streaming page: remove links to editor/dashboard 
A little security through obscurity.

Well, more like just not leading people to stuff that's not helpful to them.
Once we have auth working it shouldn't matter. Worst they can do is download expensive cuts.
 5 years ago
Mike Lang e4706232ad cutter: Fix bug in handling cancellation before finalize 
it would always think it's finalized
 5 years ago
Mike Lang 4c0e20983c Local upload backend: Always use .ts extension 
Now that we're doing that for both fast and full cuts.
 5 years ago
Mike Lang 1cb4d41334 cutter: Set upload time when marking video as done without transcoding 5 years ago
Mike Lang a7ac264201 cutter: Fix local upload backend, which was returning video id as a uuid 
instead of a string
 5 years ago
Mike Lang 5921066ee0 docker-compose: Pass node name into cutter 
Otherwise it doesn't have a stable node name (hostname inside containers
is random) and this results in orphaned videos on unclean shutdown.
 5 years ago
Mike Lang da58021c6f cutter: Fix typo where encoding settings had wrong name 5 years ago
Mike Lang 8739a7a24a thrimbletrimmer: Popup on save draft 
so we know it succeeded
 5 years ago
Mike Lang 2342b66db0 thrimbletrimmer: Don't open advanced pane if upload_location is null 5 years ago
Mike Lang e95957c54f thrimbletrimmer: Widen trim time textboxes to fit timestamps 
This might be because I'm not using webfonts or some malarkey.
 5 years ago
Mike Lang 4f6f4cad8b sheetsync: Fix typos with metrics 5 years ago
Mike Lang 2554b72c3a Fix slightly wrong default sheet name 5 years ago
Mike Lang 6724027e5a segment_coverage: Reduce missing channel/quality to warning 5 years ago
Mike Lang 4d5157cdb5 Fix a mistake with allowing reuse of name in @timed() 5 years ago
Mike Lang 1ef15780df cutter: Fix lack of info in error handling 5 years ago
Mike Lang 71333cf826 backfiller: Only run one manager, not one per channel 
Then treat backfilling each channel just like backfilling each quality.

This is conceptually simpler (only one kind of thing, a (channel, quality))
and has better behaviour when a node is down (we only have one lot of error handling around it).

It also means we aren't asking the database for the same info once per channel,
and cuts down on logging noise.
 5 years ago
Mike Lang a6cd07077a docker-compose: Wrap DSN values in quotes so spaces are allowed 5 years ago
Mike Lang ff18c7df54 backfiller: Fix issue with tracking metrics after get_nodes() failure 
There isn't an easy, clean way to pass in the DB hostname there,
and neither label is very valuable. Let's just drop it entirely.
 5 years ago
Mike Lang 6c6c1ae637 downloader: Make a few things quieter for non-important channels 5 years ago
Mike Lang 426b1328be Fix mistakes in common.requests 5 years ago
Mike Lang c1cb7966af cutter: Default to full cuts, not fast 5 years ago
Mike Lang b2a07ef114
Merge pull request #140 from ekimekim/mike/build-improvements 
Refactor dockerfiles for more shared layers
 5 years ago
Mike Lang 9e6cd71026
Merge pull request #137 from ekimekim/mike/downloader/less-important-channels 
downloader: Create concept of an "important" channel
 5 years ago
Mike Lang 8d3c863a49
Merge pull request #138 from ekimekim/mike/thrimbletrimmer/cancel-upload 
Allow thrimshim to safely cancel a job while it is cutting
 5 years ago
Mike Lang 679f1f7408
Merge pull request #139 from ekimekim/mike/restreamer/cut-types 
restreamer: Add more options for fetching cuts
 5 years ago
Mike Lang 751472217b Default to using the current commit, instead of latest 
This is generally what you want, and will behave better than using latest.
 5 years ago
Mike Lang 731ef9e2d0 Refactor dockerfiles for more shared layers 
By carefully ensuring most of our dockerfiles are identical in their first few layers,
we only need to build those layers once instead of every time.

In particular, we move installing gevent to before installing common,
so that even when common changes gevent doesn't need to be reinstalled.

This is important because gevent takes ages to install.

Also fixes segment_coverage, which wasn't being installed.
 5 years ago
Mike Lang a7f5d1c545 Fix issues with metrics gathering for cut functions 
* Need to allow timed() to have multiple callers with same name
* "type" label is reserved, use "cut_type" instead
 5 years ago
Mike Lang 58d09b419c thrimbletrimmer: Add option to select cut type for Download button 
In most cases, you want "rough" because you're going to trim the resulting
video later anyway.
 5 years ago
Mike Lang eb4fb5a9e1 restreamer: Add more options for fetching cuts 
Split full cut into two types - an mpegts one and an mp4 one.
Add "rough" cut which is just a concat of the segments.
 5 years ago
Mike Lang cc48f17348 thrimbletrimmer: Add cancel row button using the new thrimshim functionality 5 years ago
Mike Lang 3dfe0e8722 Allow thrimshim to safely cancel a job while it is cutting 
This differs from the existing reset row by only suceeding if the upload is not
in finalizing.

We also make some changes to cutter to handle this situation gracefully.
 5 years ago
Mike Lang b2cd0fa6f7 docker-compose: Support channel importance markers 
Only pass them to downloader, strip them otherwise.
 5 years ago
Mike Lang 83750da37b downloader: Create concept of an "important" channel 
In our usage, we have one channel where we really care / want to know if it's down,
but also a bunch of other channels where they're expected to not be streaming most/all of the time.

To prevent these extra channels making a ton of noise, we introduce the concept of an "important"
channel, indicated by appending a '!' to the channel name in the command line.

So for example, you might specify channels as "foo! foo_backup foo_behindthescenes".

Important channels have the same behaviour as previously.

Non-important channels:
* Have a 20-second retry on a master playlist fetch failure, instead of 5
* Log at debug when the stream is down, instead of info.
 5 years ago
Mike Lang b27e06d068 Fix typo in common/common/segments.py 5 years ago
Mike Lang 0305c0797f
Merge pull request #136 from ekimekim/mike/cut-maybe-stream 
full cut: Optionally use seekable file OR directly stream
 5 years ago
Mike Lang 4d52b18b04 cutter,restreamer: Set stream=True for full cuts when appropriate 
And also default to a new ffmpeg encoding setting for high-quality mpegts
(ie. still streamable) that is encoded very quickly.
 5 years ago