henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
262 Commits
34 Branches
0 Tags
19 MiB
Commit Graph

262 Commits (57962c8104f7376b8b38504e13e423e5b50dfe05)
 

Author SHA1 Message Date
Christopher Usher 57962c8104 allow nodes and editors to be read from csv files on start up
Christopher Usher 0eefdb94e0 docker compose updates; option not to replicate
Christopher Usher 38801ea6d3 Replication working
Christopher Usher 37b0efb66a postgres now needs building
Chris Usher a20f65d70f moving stuff around
Christopher Usher 413c86d908 script to read in dumps
Christopher Usher 1a7d3e6ff1 script for dumpting the database
Christopher Usher 84270c02ec logging fix
Christopher Usher 4d67adb277 editing a comment
Christopher Usher 67100c4126 comments
Christopher Usher 7a576c7a1f comments in docker-compose
Christopher Usher fdb5d20db7 fix to database logging
Christopher Usher 497845f2da typos in comments
Christopher Usher 14d5c41167 some small edits to docker compose
Christopher Usher 54aa919131 Update readme
Christopher Usher 72ab291a13 Update INSTALL.md
Christopher Usher b4ce8fdbb6 Update INSTALL.md
Christopher Usher 56c9a68c82 Draft of an installation guide for the wubloader
Christopher Usher 76bc629720 moved flask monitoring to its own module
Christopher Usher 73541f852f logging and monitoring for thrimshim
Christopher Usher 6c633df3ee move restreamer.stats to common.stats
Christopher Usher 6858c2e2de starting on logging and monitoring
Mike Lang 1721fbd92e fix dashboards for channel/quality naming
Mike Lang 04ef0d3823 fix a few remaining usages of StreamWorker.stream instead of .quality
Christopher Usher 361e577474 fixes based on ekimekims suggestions
Christopher Usher 732c56d502 typo in a comment
Christopher Usher 3564643613 refactoring downloader
Christopher Usher b959853593 refactored to channel and quality
Christopher Usher 720684a388 refactoring to have consistent terminology
Christopher Usher 6d38250674 starting to refactor stream to channel and variant to quality
Mike Lang a2b21966b9
Merge pull request from ekimekim/mike/dashboards 
Add grafana dashboards as jsonnet code
Mike Lang f7b591e78b sheetsync: Log more information on HTTPError 
The api gives additional detail that we want to know when debugging.
Mike Lang 73d5941e05 downloader: Track timestamp of latest segment 
This gives us a "stream delay" metric.

Prom doesn't have any native way to check the current value of a metric,
in order to take max(). It only offers increment and set.

We reach into some internals to do this in a hacky way,
but the cleaner way would be to track the value ourselves and have a prom callback
that gets the value.

Sigh, I hate this prom library. I might write my own that's less dumb.
Mike Lang e4d3e418c8 transcode checker: longer retry while waiting for videos to finish 
but still check db often.
This prevents us from using too much api quota on these checks,
while still letting us spot new videos quickly.
Mike Lang 1f15900b6f cutter: At least for now, don't auto-retry errors 
This leads to rapidly exhausting our upload limit since even a fast failed request
costs the same amount of usage quota as a 1-hour long video.
Mike Lang fbef4725d7 cutter: Handle case where we are told to stop while looking for candidates 
Previously, it would return None and things would break. Now the None is handled
correctly, and is documented.
Mike Lang 5cec6ec96e cutter: Reconnect after any error that might be a database error 
After certain kinds of DB error (eg. lost conn), we need to make a new conn
to have things work again. To be safe, we just do it after every error where it might
be a problem.
Mike Lang fea9ff6c1d cutter: Fix dockerfile, which was missing ffmpeg dependency
Mike Lang f50276bd01 backfiller: Expose recent_cutoff as CLI arg and increase it to 120s default 
In testing, GDQ's stream delay went up over 1min, which caused backfillers to backfill
segments at the same time they were downloaded. We increase the window for now,
and also make it configurable.
MasterGunner 6fa9d9d388
Merge pull request from ekimekim/gunner/additional-thrimbletrimmer-integration 
Gunner/additional thrimbletrimmer integration
MasterGunner 2e953eddde Cleanup from Ekim's comments, removed auth placeholder until I know what I'm doing.
Mike Lang ca925ae2e6 dashboard: Add some extra detail sections for backfiller and downloader
MasterGunner 6a171130e8 Updated Get All Rows route.
Mike Lang 39e7a5c2e6 Add overview dashboard
MasterGunner 736f0e0fe4 Adding get_all_row and auth function stubs
Mike Lang 41fffc2809
Merge pull request from ekimekim/mike/monitoring 
Scripts for running prometheus/grafana for monitoring
Mike Lang 612e34b88d
Merge pull request from ekimekim/mike/backfiller/concurrent 
backfiller: Allow multiple concurrent segment downloads
Mike Lang 29040a166c backfiller: Allow multiple concurrent segment downloads 
This will signifigantly increase throughput when downloading
large ranges of segments.

The max concurrency is exposed as a cli arg.

We also slightly modify the logged info, so it reports segments downloaded,
not just number of missing segments (which we might skip downloading for various reasons).
Christopher Usher ec5a545fd2 Merge branch 'mike/sheetsync/fix-db-error'
Mike Lang 7273ee071e monitoring fixes