Commit Graph

308 Commits (24e4e0ba28b34bea444b760b0491dacb9b44acd2)
 

Author SHA1 Message Date
Christopher Usher 365d7b7852 Placeholder text 5 years ago
Christopher Usher ea42cab8fd Added editor, edit_time and upload_time to thrimshim and cutter updates of the
database
5 years ago
Christopher Usher fb889cd078 database changes to keep track of editors and edit times 5 years ago
Christopher Usher 84270c02ec logging fix 5 years ago
Christopher Usher 4d67adb277 editing a comment 5 years ago
Christopher Usher 67100c4126 comments 5 years ago
Christopher Usher 7a576c7a1f comments in docker-compose 5 years ago
Christopher Usher fdb5d20db7 fix to database logging 5 years ago
Christopher Usher 497845f2da typos in comments 5 years ago
Christopher Usher 14d5c41167 some small edits to docker compose 5 years ago
Christopher Usher 54aa919131 Update readme 5 years ago
Christopher Usher 72ab291a13 Update INSTALL.md 5 years ago
Christopher Usher b4ce8fdbb6 Update INSTALL.md 5 years ago
Christopher Usher 56c9a68c82 Draft of an installation guide for the wubloader 5 years ago
Christopher Usher 76bc629720 moved flask monitoring to its own module 5 years ago
Christopher Usher 73541f852f logging and monitoring for thrimshim 5 years ago
Christopher Usher 6c633df3ee move restreamer.stats to common.stats 5 years ago
Christopher Usher 6858c2e2de starting on logging and monitoring 5 years ago
Mike Lang 1721fbd92e fix dashboards for channel/quality naming 5 years ago
Mike Lang 04ef0d3823 fix a few remaining usages of StreamWorker.stream instead of .quality 5 years ago
Christopher Usher 361e577474 fixes based on ekimekims suggestions 5 years ago
Christopher Usher 732c56d502 typo in a comment 5 years ago
Christopher Usher 3564643613 refactoring downloader 5 years ago
Christopher Usher b959853593 refactored to channel and quality 5 years ago
Christopher Usher 720684a388 refactoring to have consistent terminology 5 years ago
Christopher Usher 6d38250674 starting to refactor stream to channel and variant to quality 5 years ago
Mike Lang a2b21966b9
Merge pull request #65 from ekimekim/mike/dashboards
Add grafana dashboards as jsonnet code
5 years ago
Mike Lang f7b591e78b sheetsync: Log more information on HTTPError
The api gives additional detail that we want to know when debugging.
5 years ago
Mike Lang 73d5941e05 downloader: Track timestamp of latest segment
This gives us a "stream delay" metric.

Prom doesn't have any native way to check the current value of a metric,
in order to take max(). It only offers increment and set.

We reach into some internals to do this in a hacky way,
but the cleaner way would be to track the value ourselves and have a prom callback
that gets the value.

Sigh, I hate this prom library. I might write my own that's less dumb.
5 years ago
Mike Lang e4d3e418c8 transcode checker: longer retry while waiting for videos to finish
but still check db often.
This prevents us from using too much api quota on these checks,
while still letting us spot new videos quickly.
5 years ago
Mike Lang 1f15900b6f cutter: At least for now, don't auto-retry errors
This leads to rapidly exhausting our upload limit since even a fast failed request
costs the same amount of usage quota as a 1-hour long video.
5 years ago
Mike Lang fbef4725d7 cutter: Handle case where we are told to stop while looking for candidates
Previously, it would return None and things would break. Now the None is handled
correctly, and is documented.
5 years ago
Mike Lang 5cec6ec96e cutter: Reconnect after any error that might be a database error
After certain kinds of DB error (eg. lost conn), we need to make a new conn
to have things work again. To be safe, we just do it after every error where it might
be a problem.
5 years ago
Mike Lang fea9ff6c1d cutter: Fix dockerfile, which was missing ffmpeg dependency 5 years ago
Mike Lang f50276bd01 backfiller: Expose recent_cutoff as CLI arg and increase it to 120s default
In testing, GDQ's stream delay went up over 1min, which caused backfillers to backfill
segments at the same time they were downloaded. We increase the window for now,
and also make it configurable.
5 years ago
MasterGunner 6fa9d9d388
Merge pull request #64 from ekimekim/gunner/additional-thrimbletrimmer-integration
Gunner/additional thrimbletrimmer integration
5 years ago
MasterGunner 2e953eddde Cleanup from Ekim's comments, removed auth placeholder until I know what I'm doing. 5 years ago
Mike Lang ca925ae2e6 dashboard: Add some extra detail sections for backfiller and downloader 5 years ago
MasterGunner 6a171130e8 Updated Get All Rows route. 5 years ago
Mike Lang 39e7a5c2e6 Add overview dashboard 5 years ago
MasterGunner 736f0e0fe4 Adding get_all_row and auth function stubs 5 years ago
Mike Lang 41fffc2809
Merge pull request #62 from ekimekim/mike/monitoring
Scripts for running prometheus/grafana for monitoring
5 years ago
Mike Lang 612e34b88d
Merge pull request #61 from ekimekim/mike/backfiller/concurrent
backfiller: Allow multiple concurrent segment downloads
5 years ago
Mike Lang 29040a166c backfiller: Allow multiple concurrent segment downloads
This will signifigantly increase throughput when downloading
large ranges of segments.

The max concurrency is exposed as a cli arg.

We also slightly modify the logged info, so it reports segments downloaded,
not just number of missing segments (which we might skip downloading for various reasons).
5 years ago
Christopher Usher ec5a545fd2 Merge branch 'mike/sheetsync/fix-db-error' 5 years ago
Mike Lang 7273ee071e monitoring fixes 5 years ago
Mike Lang 5a6d443efd grafana: View-only anonymous access 5 years ago
Christopher Usher 980875b6f3 Merge branch 'mike/sheetsync/fix-db-error' of https://github.com/ekimekim/wubloader into mike/sheetsync/fix-db-error 5 years ago
Christopher Usher 37bad7d5ed Also reset database connection on error in the backfiller 5 years ago
Christopher Usher 28f350dd46 Also reset database connection on error in the backfiller 5 years ago