henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
1,450 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'bd89655190'
${ noResults }
Commit Graph

1450 Commits (bd89655190ab1e5816101a6dbcd0b1707162fa4e)
 

Author SHA1 Message Date
Mike Lang 3fb6676fbe pubbot: Add more detail to logging 
Wrap raw pubnub messages in a metadata object containing:
- time
- our hostname
- our pid
- any inferred information

Plus make a message-less log write on startup so we know where there might be gaps.

Also do more error handling so that we don't completely lose messages on error.
 2 months ago
Mike Lang 9b19c12a53 Add pubnub-watching zulip bot 
Which also records all pubnub messages for archival to a file SEGMENTS/pubnub-log.json.
We ran this for most of last year, but not as part of wubloader.
 2 months ago
ZeldaZach c378a1e4ab Add Audit Logging for several endpoints 
- Use transactions for DB commits to avoid audit-less logs

Endpoints Supported:
- Manual Link
- Reset Row
- Update Row
 2 months ago
Mike Lang fd78ff288e fix bugs in postgres schema from earlier changes 2 months ago
Mike Lang 7483794a23 docker-compose: Set up sheetsync for downloading media links 
This involves giving it access to the SEGMENTS/media directory.
 2 months ago
Mike Lang 3e7cb38cf0 sheetsync: Optionally download media linked in image links column 
To enable this, you need to:
- set --media-dir globally for sheetsync
- enable download_media=true for the events sync config
To disable for individual rows (eg. because of known issues), put "[nodownload]" in the notes column.
 2 months ago
Mike Lang 7b590cf574 chat-archiver: Some cleanups to the URL matching regex 
With thanks to Me-Me for review
 2 months ago
Mike Lang 15f86551d4 docker-compose: Backfill of media, with or without chat archiver 2 months ago
Mike Lang 9dfb00f4ab chat_archiver: Logic for checking and downloading media links 2 months ago
Mike Lang 2855ec759d download_media: Add pdf to default allowed content types 
We want to capture linked PDFs in addition to videos and images
 2 months ago
Mike Lang b46c577014 download_media: Add function for checking if a URL has been downloaded before 2 months ago
Mike Lang 352c9e9081 download_media: Get data from potentially malicious URLs and store in the filesystem 
This is suitable for taking arbitary URLs from chat, etc and trying to fetch them.
It downloads them to a filepath that contains a hash of the URL and content.
 2 months ago
Mike Lang 07055e3605 chat-archiver: extract the ensure_emotes greenlet management to a class 2 months ago
Mike Lang 3f6263a037 playlist-manager: More fixes 3 months ago
Mike Lang d8953568b2 playlist_manager: Add --once option 3 months ago
Mike Lang 9d12d37053 playlist_manager: Fixes 3 months ago
Mike Lang 23ad78d592 Record in database when end time is "--" 
We need this so that reverse sync reproduces these values correctly.

To handle this in the database, we have a composite type (dashed: boolean, value: timestamp).
Value is always valid and is equivalent to the old timestamp column,
but must be equal to start_time if dashed is true.

The only place we directly reference this column outside sheetsync is thrimshim, where we
always consider the value only.
 3 months ago
Mike Lang e50adbf2da Fix a bug where transitions past the first are not timed correctly 
The video offset is timed relative to the full video up until that point, not the previous range.
 3 months ago
Mike Lang 69bfa79e24 Make video transition durations floats, not intervals 
Almost all code was already expecting this.
 3 months ago
Mike Lang 649a0fb2dd cutter: Improve error message for unhandled cutting exceptions 
In particular, use exception chaining instead of printing the old error.
 3 months ago
Mike Lang 55f48e1881 cutter: Add unlisted-only safety flag to youtube upload backend 3 months ago
Mike Lang 12150a4005 thrimshim: Return list of available transitions 
For use in thrimbletrimmer for a drop-down.
 3 months ago
Mike Lang d4de1f94be Add descriptions to xfade transitions 3 months ago
Mike Lang 1dec53924f fix typo 3 months ago
Mike Lang 91511295c0 Fix poster moment tag 3 months ago
Mike Lang 25e5e933b5 Fix typo 3 months ago
Mike Lang 7d89569ead Code more defensively around out-of-order or missing parent ids 3 months ago
Mike Lang 760dbd1e07 sheetsync: Convey row parent info 
by prefixing with a number of ^ characters and noting the parent id in the notes column.
 3 months ago
Mike Lang 23960d947b streamlog: implicit tags must be first 3 months ago
Mike Lang 664f98150f sheets implicit tags fix 3 months ago
Mike Lang 62491d119f debugging 3 months ago
Mike Lang 06931db26b more sheetsync fixes 3 months ago
Mike Lang 1a5687204d debugging 3 months ago
Mike Lang 820f7322f9 more streamlog fixes 3 months ago
Matija Rezar fbb6c4dca0 current stat dump 3 months ago
Mike Lang b68abb92e2 more streamlog column names 3 months ago
Mike Lang 484e768303 correct streamlog column name 3 months ago
Mike Lang 20ee382b7f sheetsync: Fix missing arg 
Just hard-code that the tags sheet is always called "Tags" for now, we can fix later
if it ever needs to change.
 3 months ago
Mike Lang c6058cbdd0 sheetsync: Update streamlog playlist middleware with API changes 
To allow setting playlist name and show_in_description
 3 months ago
Mike Lang 367e6a7a7a sheetsync fixes 
typos and omissions
 3 months ago
Mike Lang d5f73c226c GoogleAPIClient: Improve error reporting 3 months ago
Mike Lang 227be23cab postgres setup: split schema into its own file 
This is more convenient for development since it's a standalone file that you can
apply without using the script, it also means we get syntax highlighting.
 3 months ago
Mike Lang 48725ac739 docker-compose: fix typos 3 months ago
Mike Lang 44f4433cad more typos in streamlog archive middleware 3 months ago
Mike Lang eeffeeed10 sheetsync: Deal with reverse syncing properly when not all events are in the list of worksheets 
This is important because archive events should not be reversed.
We only want to create new rows when the row's intended worksheet is in our list of worksheets we sync.
 3 months ago
Mike Lang 96181fd875 Support archive sync in sheetsync again 3 months ago
Mike Lang a3aaa37bb0 Restructure playlists table 
to have all tags, not just playlists.
Notably, this means playlist ids may be null.
 3 months ago
Mike Lang 145ddfc7ff Fix some typos in StreamLogPlaylistsMiddleware 3 months ago
Mike Lang 0f06f96a7c Fix: Move events-specific observe_rows() out of SheetSync 3 months ago
Mike Lang 8dd6741ee9 schema: Playlist tags may be null, indicating "match nothing" 
This is intended for partially-entered playlist data where tags have not been specified yet.
 3 months ago