henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
900 Commits
24 Branches
0 Tags
15 MiB
master
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'b6a2d7e16a'
${ noResults }
Commit Graph

900 Commits (b6a2d7e16a00a08492abbda7c38e58cc8fca5474)
 

Author SHA1 Message Date
ElementalAlchemist b6a2d7e16a Give the thumbnail options controls some room to breathe 2 years ago
ElementalAlchemist 8c872f6999 Add image preview generation 2 years ago
ElementalAlchemist 9fa898d5aa Fix doing the time conversion before the data required for the time conversion is loaded 2 years ago
ElementalAlchemist 0e787677a0 Load thumbnail data for video 2 years ago
ElementalAlchemist 64f0ac714f Set up thumbnail editing with data plumbing through 2 years ago
Mike Lang 46ad45bdb9 thrimshim: Don't 500 if MODIFIED video doesn't contain some fields 
Missing fields _should_ be interpreted as "don't update these" but we weren't doing that.
 2 years ago
Mike Lang ab985cf1b0 Fix syntax error 2 years ago
Mike Lang 57d89b9b39 Fix missing imports 2 years ago
Mike Lang 05816ac51d fix a bug where shift is calculated wrong 
we need to use the hour in PST, which we just hard-code as 8 hours.
If we ever hit DST before the run, we'll go back and do it properly.
 2 years ago
Mike Lang cfe93e31dd Fix a bug where modified videos are missing static tags 2 years ago
Mike Lang bd519369bd thumbnail fixes 2 years ago
Mike Lang 31c3f0b8b1 OS-level dependencies for Pillow 2 years ago
Mike Lang 24c066e3e8 Add thumbnail support to thrimshim 
All thumbnail columns are modifiable, so you can eg. switch from
a templated thumbnail to a custom one, or change the time and template.
 2 years ago
Mike Lang 0f0aee36b3 restreamer: Add endpoint for previewing thumbnails 2 years ago
Mike Lang 45c46df8bb Add thumbnail templating code 2 years ago
Mike Lang d3e21ae9b0 Implement thumbnails in cutter 2 years ago
Mike Lang ce1f50db06 Database changes for thumbnails 2 years ago
Mike Lang 0681902789 Trigger download of the image file in a hacky way 
Apparently this is the Recommended Approach for doing this - you create an <a> element,
set its attributes, then temporarily add it to the page and click() it.

The downloads.download() api we found earlier is exclusive to Chrome Apps.
 2 years ago
Mike Lang adb6e2ae10 thrimbletrimmer: Add button to download current frame 
It always uses source quality and downloads as a PNG.

We use the browser.downloads api to construct the URL on demand.

Note we might not always get the exact right frame in 60fps streams,
we might get one before or after (and "frame seek" in the editor skips 2 frames in this case).
 2 years ago
ElementalAlchemist 1cb819a4c5 Use more native HTML options for editor keyboard shortcuts help 2 years ago
Mike Lang ffae321d04 editor: Make public checkbox "unlisted" instead, with negated meaning 
less confusing
 2 years ago
Mike Lang b9c44375c3 playlist_manager: Don't add non-public videos to playlists 
So that playlists don't pick up unlisted videos.
When the video is modified to be public, it will be added as normal.

However, note that since playlist_manager never removes videos from playlists,
making an existing video public video unlisted will not remove it from playlists.
 2 years ago
Mike Lang 36017aaccd sheetsync: Show unlisted videos in DONE state as UNLISTED instead 
We don't actually want to represent them as a different state in the backend, but showing
them differently on the sheet is helpful to humans.
 2 years ago
Mike Lang bd948235fb thrimbletrimmer: Add advanced submission option for "public" videos 
Note that the default is true, so we only expand the advanced pane if it's false.
False corresponds to uploading a video as "unlisted".

Also mark it as modifyable.
 2 years ago
Mike Lang 46468409cb Add "public" column to control if videos are unlisted 
This column is modifyable.
This replaces the old "hidden" argument to youtube upload backends.
 2 years ago
Mike Lang 50b830bf82 github actions: check out submodules when building 2 years ago
Mike Lang 4f2c5c48ec build chat_archiver in CI 2 years ago
Mike Lang 08257386e2 Add restreamer endpoint for viewing chat messages 2 years ago
Mike Lang 9320251de7 Some extra documentation on chat_archiver 2 years ago
Mike Lang d8a9b5ddf0 chat_archiver: Always sort json object keys to ensure canonical output 2 years ago
Mike Lang 651658e507 JOINs and PARTs have been observed with up to 30sec difference 
it turns out to be completely undocumented what the max delay is. so let's assume 45s.
anything > 60s might cause problems due to matching messages being more than 1 batch apart.
 2 years ago
Mike Lang c50224415c more backfiller chat fixes 
fixup: more backfiller fixes

Enable backfilling of chat logs
 2 years ago
Mike Lang 05a989f67d chat-archiver: fixes 2 years ago
Mike Lang c1c1c11bce chat_archiver: Add prometheus metrics 2 years ago
Mike Lang c25d79e7c2 chat-archiver: Merge all files every minute 2 years ago
Mike Lang 4cfc362f76 chat-archiver: pass in node name 
instead of using container hostname
 2 years ago
Mike Lang a48beab576 chat-archiver: update girc for py3 support and fixes 2 years ago
Mike Lang 315c9c8297 Integrate chat archiver as a proper component 2 years ago
Mike Lang f8b3ace148 Backfill chat archives under the "chat" quality 2 years ago
Mike Lang 05ddd39504 chat_archiver: Split files into directories by hour 
matching how we handle video files
 2 years ago
Mike Lang 1d626738bd chat_archiver: Start a new client on RECONNECT 2 years ago
Mike Lang 96cc212bf0 chat_archiver: fixes, implement merge_all 2 years ago
Mike Lang d32cbbb7e1 chat-archiver: File merging and other fixes 2 years ago
Mike Lang 0756539b85 chat-archiver: Early work and basic archival 2 years ago
Mike Lang 6b9d8ab785 Fix submit changes button - should be hidden by default 2 years ago
Mike Lang ac6612fb85 Add ability to submit video as MODIFIED to thrimbletrimmer 2 years ago
Mike Lang f06be1f391 thrimshim changes for modified 
would be ideal to check all args to see if any don't match (and refuse to modify in that case)
but eh too much work to properly normalise.
 2 years ago
Mike Lang c4a1d72240 Add new state to postgres and implement cutter logic 2 years ago
Mike Lang 1add3c5c22 Implement tombstoning to allow for segment deletion 
Rarely, we find ourselves needing to explicitly delete some data, eg. something that shouldn't
have been public and should be removed from all records.

It would also be nice if we could "clean up" bad versions of the same segment,
which occasionally come up when downloaders have issues.

With our distributed segment database, this is actually rather difficult as deleting the data
from any one server would cause it to be restored from the others. It was only possible
by stopping all backfill, deleting the data on all servers, then starting backfill again.

Here we introduce a more practical approach. An operator creates an empty flag file
with the same name as the segment to be deleted, but with a `.tombstone` extension.
eg. to delete a file `/segments/desertbus/source/2019-11-13T02/45:51.608000-2.0-full-7IS92rssMzoSBQDIevHStbTNy-URRV3Vw-jzZ6pwOZM.ts`,
you would create a tombstone `/segments/desertbus/source/2019-11-13T02/45:51.608000-2.0-full-7IS92rssMzoSBQDIevHStbTNy-URRV3Vw-jzZ6pwOZM.tombstone`.

These tombstone files do two important things:
* They hide the segment from being listed, which both means:
  * It can't be restreamed or put into a video
  * It can't be backfilled to other nodes
* The tombstone files themselves do get backfilled to other nodes, so you only need to mark them on one server.

Once the tombstone has propagated to all nodes, the segment file can be deleted independently on each one.

We chose not to have a tombstone automatically trigger a segment deletion for safety reasons.
 2 years ago
Mike Lang 44d0c0269a cache results of common.segments.best_segments_by_start 
The restreamer spends most of its time iterating through segments (parsing them, determining the best one for each start time)
to serve large time ranges. Since this only depends on the list of filenames read from disk,
we can cache it for a given hour as long as that list is identical.

This is a little trickier than it sounds because best_segments_by_start is an iterator
and in most cases it won't be fully consumed. So we introduce a `CachedIterator` abstraction
that will both remember the previously yielded values, and keep track of the live iterator
so it can be resumed again if a previous invocation only partially consumed it.

This also has the nice side effect of merging simultaneous operations - if two requests come in
for the same hour at the same time, they'll share one iterator and both consume the results
as they come in.
 2 years ago