henine
/
wubloader
mirror of https://github.com/ekimekim/wubloader
1
0
Fork 0
Code Issues Releases Wiki Activity
385 Commits
34 Branches
0 Tags
19 MiB
thrimbletrimmer-solid
HubbeKing-multiplatform-builds
bus-progress-ui
master
bus-post-processing
chrusher/bus_synthesizer
mike/faster-chat-merge
mike/chat-no-end-time
chrusher-playlist-thumbnails
chrusher-thumbnail-frontend
mike/prizebot
condor-scripts
mike/chat-subs
mike/bad-segment-hack
mike/backfill-extras
mike/cropping
mike/shorts
mike/download-frame-2
mike/debug-playlists
chrusher/backfiller_error_handler
sheet-updates
mike/generate-videos
suspect-segments
mike/testing
gunner/thrimbletrimmer-improvements
mike/render-dashboard
mike/database/initial
downloader/follow-game
mike/downloader/conn-pooling
mike/common/split-up
mike/backdoor
mike/downloader/handle-404
mike/non-restreamer-stats
archive/original-v2.1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'e5c5eb4090'
${ noResults }
Commit Graph

385 Commits (e5c5eb4090b0a00457c48b1eadea7e9140253541)
 

Author SHA1 Message Date
Mike Lang a2b21966b9
Merge pull request #65 from ekimekim/mike/dashboards 
Add grafana dashboards as jsonnet code
 5 years ago
Mike Lang f7b591e78b sheetsync: Log more information on HTTPError 
The api gives additional detail that we want to know when debugging.
 5 years ago
Mike Lang 73d5941e05 downloader: Track timestamp of latest segment 
This gives us a "stream delay" metric.

Prom doesn't have any native way to check the current value of a metric,
in order to take max(). It only offers increment and set.

We reach into some internals to do this in a hacky way,
but the cleaner way would be to track the value ourselves and have a prom callback
that gets the value.

Sigh, I hate this prom library. I might write my own that's less dumb.
 5 years ago
Mike Lang e4d3e418c8 transcode checker: longer retry while waiting for videos to finish 
but still check db often.
This prevents us from using too much api quota on these checks,
while still letting us spot new videos quickly.
 5 years ago
Mike Lang 1f15900b6f cutter: At least for now, don't auto-retry errors 
This leads to rapidly exhausting our upload limit since even a fast failed request
costs the same amount of usage quota as a 1-hour long video.
 5 years ago
Mike Lang fbef4725d7 cutter: Handle case where we are told to stop while looking for candidates 
Previously, it would return None and things would break. Now the None is handled
correctly, and is documented.
 5 years ago
Mike Lang 5cec6ec96e cutter: Reconnect after any error that might be a database error 
After certain kinds of DB error (eg. lost conn), we need to make a new conn
to have things work again. To be safe, we just do it after every error where it might
be a problem.
 5 years ago
Mike Lang fea9ff6c1d cutter: Fix dockerfile, which was missing ffmpeg dependency 5 years ago
Mike Lang f50276bd01 backfiller: Expose recent_cutoff as CLI arg and increase it to 120s default 
In testing, GDQ's stream delay went up over 1min, which caused backfillers to backfill
segments at the same time they were downloaded. We increase the window for now,
and also make it configurable.
 5 years ago
MasterGunner 6fa9d9d388
Merge pull request #64 from ekimekim/gunner/additional-thrimbletrimmer-integration 
Gunner/additional thrimbletrimmer integration
 5 years ago
MasterGunner 2e953eddde Cleanup from Ekim's comments, removed auth placeholder until I know what I'm doing. 5 years ago
Mike Lang ca925ae2e6 dashboard: Add some extra detail sections for backfiller and downloader 5 years ago
MasterGunner 6a171130e8 Updated Get All Rows route. 5 years ago
Mike Lang 39e7a5c2e6 Add overview dashboard 5 years ago
MasterGunner 736f0e0fe4 Adding get_all_row and auth function stubs 5 years ago
Mike Lang 41fffc2809
Merge pull request #62 from ekimekim/mike/monitoring 
Scripts for running prometheus/grafana for monitoring
 5 years ago
MasterGunner 4423ddee3c
Update SecurityModel.md 
Simplified the document based on our discussions.
 5 years ago
Mike Lang 612e34b88d
Merge pull request #61 from ekimekim/mike/backfiller/concurrent 
backfiller: Allow multiple concurrent segment downloads
 5 years ago
Mike Lang 29040a166c backfiller: Allow multiple concurrent segment downloads 
This will signifigantly increase throughput when downloading
large ranges of segments.

The max concurrency is exposed as a cli arg.

We also slightly modify the logged info, so it reports segments downloaded,
not just number of missing segments (which we might skip downloading for various reasons).
 6 years ago
Christopher Usher ec5a545fd2 Merge branch 'mike/sheetsync/fix-db-error' 6 years ago
Mike Lang 7273ee071e monitoring fixes 6 years ago
Mike Lang 5a6d443efd grafana: View-only anonymous access 6 years ago
Christopher Usher 980875b6f3 Merge branch 'mike/sheetsync/fix-db-error' of https://github.com/ekimekim/wubloader into mike/sheetsync/fix-db-error 6 years ago
Christopher Usher 37bad7d5ed Also reset database connection on error in the backfiller 6 years ago
Christopher Usher 28f350dd46 Also reset database connection on error in the backfiller 6 years ago
Mike Lang e048db0d94 cutter: Fix a failure mode where we never recover from a DB conn failure in TranscodeChecker 
Since we never got a new conn after failure, we would just keep erroring with
"connection already closed" errors.

This isn't applicable to the main cutter loops since a DB failure there will restart the process.
 6 years ago
Mike Lang fe68e98804 sheetsync: Fix a failure mode where we never recover from a DB conn failure 
Since we never got a new conn after failure, we would just keep erroring with
"connection already closed" errors.
 6 years ago
Mike Lang a767760f02 Add some existing scripts for setting up prometheus 6 years ago
Mike Lang 90eb2a4f13
Merge pull request #59 from ekimekim/mike/fixes 
Some misc fixes from cutter and backfiller, see commits
 6 years ago
Mike Lang 51efeb1f12
Merge pull request #58 from ekimekim/mike/nginx-dns-hack 
Fix nginx when some services are disabled
 6 years ago
Mike Lang 7179fcacec Backfiller: ignore temp segments 
To make this work, we make type a proper segment field.

We also tell get_best_segments to ignore temp segments, since they might go away
before we can actually use them.
 6 years ago
Mike Lang 85c110ccb4 cutter: Fix typo from when we moved to the client model instead of auth headers 6 years ago
Mike Lang 3fa3c73d0e Fix nginx when some services are disabled 
nginx tries to resolve everything at startup, which doesn't work
if some of the services aren't present.

we instead generate the config file from a passed in env var, so that only
enabled services are present.
 6 years ago
Mike Lang 6d729fa5cc
Merge pull request #57 from ekimekim/mike/compose-bits 
Some stuff for making the docker compose setup easier
 6 years ago
Mike Lang 6071a2f18d docker_compose: Add a local postgres instance as an optional service 
The node hosting the database can then easily run it as part of the stack.
 6 years ago
Mike Lang 63eb324ba5 Add nginx service that provides a frontend to all the other services 
This allows us to run all the different services and expose all their metrics,
all on one port.
 6 years ago
Mike Lang a7a54db726 docker-compose: Restructure for some finer control 
Allow enabling/disabling at top of file
Allow no port to be exposed for any service
 6 years ago
Mike Lang 499e486b0b
Merge pull request #54 from ekimekim/mike/sheet-sync/initial 
sheet sync
 6 years ago
Mike Lang 018e920808 sheet-sync: Some fixes 6 years ago
Christopher Usher dd246e1343 ekimekim's suggestions 6 years ago
Christopher Usher 9b28765ff2 Bug fixes to get the database connection working 6 years ago
Christopher Usher 4b9fbcb7d2 backfiller database code 6 years ago
Mike Lang 6c47f32d16 docker-compose.jsonnet: Add sheet sync 6 years ago
Mike Lang f354130434 sheetsync: Only allocate ids when first needed 
This prevents rate limiting issues when immediately allocating all 999 ids
for an empty sheet.
 6 years ago
Mike Lang 11fc67f071 sheetsync: Review feedback 
* Expand on some comments
* Fix conflicting port number
* Write help text for all args
 6 years ago
Mike Lang 9762f308a0 Implement main part of sheet sync 6 years ago
Mike Lang 5a44bfdf51 Google sheets api wrapper 
Exposes a way to read all rows, and write a single cell.

We need to read all columns of each row so we know what would be modified
so we only do updates to single cells that aren't already the correct value.

This keeps us from impacting the sheet load too much with constantly changing values,
which I think might be a thing even if the values are the same.
 6 years ago
Mike Lang 3647d091f8 Move common google api auth functionality into common 
So we can reuse it for google sheets
 6 years ago
Mike Lang 2b4d2cce90 sheet sync: Basic skeleton 6 years ago
Mike Lang 3ccace2a73 database: Update constraints to allow null edit inputs in state DONE 
This allows manual uploads to work without needing to fill all the edit fields
with junk.

We also set a constraint on uploader asserting that any videos from claimed onwards have a known uploader.
Again, an exception is made for DONE to allow manual uploads.
 6 years ago