Commit Graph

65 Commits (condor-scripts)

Author SHA1 Message Date
Mike Lang c37d44f7c5 blah 3 years ago
Mike Lang e4700e6e4f refactor reviews for extra info reporting even on happy path 3 years ago
Mike Lang c1465f49a6 Fix a bug where finish range is wrong if start range doesn't start at 0
It was in the wrong direction so eg. starting 10sec early caused finish to be 20sec late
3 years ago
Mike Lang a6fffa6a72 Automatically take first found start in reviews, but warn user if we do
The vast majority of the time, this is just because of an early 1-1 trapdoor.
But we should double-check.
3 years ago
Mike Lang 95fe403b91 review: don't return an absolute location header
bad old flask version forces an absolute location when it shouldn't
3 years ago
Mike Lang 2b8bf96e62 review: improve blackdetect parsing
sometimes ffmpeg will use a bare \r before writing out the blackdetect line,
not a \n. catch this case by splitting on either.
3 years ago
Mike Lang 345cb53f4e review: allow manual start selection
* return start video
* use ubuntu instead of alpine because ffmpeg in alpine lacks drawtext support
* only cache if inputs are exactly the same, preventing accidents (prev scheme too weak)
* fix a bug where end range is wrong if start_range doesn't start at 0
	conceptually, this was a more serious bug as time_offset was wrong, but luckily the bug applied
	equally to both racers so there was no net effect
3 years ago
Mike Lang d37cca52c9 review: Avoid race conditions if called multiple times
By always using unique filepaths (except for the final result, which is written atomically)
we turn multiple simultanious calls into at worst wasted effort, instead of garbled results.
3 years ago
Mike Lang a236509470 and get it working 3 years ago
Mike Lang 04ae9dd695 Finish adapting cut_review for restreamer 3 years ago
Mike Lang 7feaa2d31d wip: unfinished: port cut_reviews.py into restreamer 4 years ago
Mike Lang 5f16ce4c3b restreamer: fix bug in /cut 4 years ago
Mike Lang 2dbb8dbaee improve replay path behaviour for missing sources and change placeholder 4 years ago
Mike Lang 6bc3b50727 wip: 4 years ago
Mike Lang e789e04b27 restreamer: Hack in a path to re-stream "last 20 seconds", for live 'replays' 4 years ago
Mike Lang efe185bc0c Fix copy-paste error in restreamer metrics endpoint 4 years ago
Mike Lang 48ef416dfb restreamer, thrimshim: Allow /metrics/* in addition to /metrics
This allows the metrics proxying from nginx to work even if the path is not rewritten,
which is hard to do in k8s.
4 years ago
HubbeKing 86f7823348 Replace calls to gevent.signal() with gevent.signal_handler()
gevent.signal() was removed in gevent 1.5a4, see http://www.gevent.org/api/gevent.signal.html
Removed on Feb 5th, see https://github.com/gevent/gevent/pull/1530
4 years ago
Mike Lang a53786dc2d Add file and make as build dependencies
gevent now requires these to build. I'm not sure when this changed.
4 years ago
Mike Lang 4d21f447ad Add restreamer call to generate full concat'd videos
This makes the timelapse easier.
5 years ago
Mike Lang b39e844c1e restreamer: Fix missing import of smart cut 5 years ago
Mike Lang b516917e62 Add new "smart" cut technique 5 years ago
Mike Lang dc7f093ba0 Disable mp4 option for restreamer cuts
It caused our RSS to explode and i'm not sure why
5 years ago
Mike Lang b2a07ef114
Merge pull request #140 from ekimekim/mike/build-improvements
Refactor dockerfiles for more shared layers
5 years ago
Mike Lang 731ef9e2d0 Refactor dockerfiles for more shared layers
By carefully ensuring most of our dockerfiles are identical in their first few layers,
we only need to build those layers once instead of every time.

In particular, we move installing gevent to before installing common,
so that even when common changes gevent doesn't need to be reinstalled.

This is important because gevent takes ages to install.

Also fixes segment_coverage, which wasn't being installed.
5 years ago
Mike Lang eb4fb5a9e1 restreamer: Add more options for fetching cuts
Split full cut into two types - an mpegts one and an mp4 one.
Add "rough" cut which is just a concat of the segments.
5 years ago
Mike Lang 4d52b18b04 cutter,restreamer: Set stream=True for full cuts when appropriate
And also default to a new ffmpeg encoding setting for high-quality mpegts
(ie. still streamable) that is encoded very quickly.
5 years ago
Mike Lang b4310b0024 restreamer: Allow blank start/end for cut
So it acts the same as generating a playlist.
5 years ago
Mike Lang 09887f17ab restreamer: Add option to download full cut instead of fast cut
Transcoding to mp4 with no specific encoding args as a reasonable default.
5 years ago
Mike Lang d3e1d6b4fc Resurrect non-experimental cut, now dubbed "full" (vs "fast") cut
In a fast cut, we edit the first and last segments then concatenate them all.
However, this leads to some tiny but perciptible artifacting around the border
of the first and second (and second-last and last) segments.

A full cut is much slower, but re-encodes the video into the desired format
and is more reliable.

We want both options to be available.

With this commit, we only add the option, we don't use it in restreamer or cutter.
5 years ago
Christopher Usher 76bc629720 moved flask monitoring to its own module 5 years ago
Christopher Usher 6c633df3ee move restreamer.stats to common.stats 5 years ago
Christopher Usher 361e577474 fixes based on ekimekims suggestions 5 years ago
Christopher Usher b959853593 refactored to channel and quality 5 years ago
Mike Lang 5c84e8dfab restreamer: Fix wrong name for parse function
derp
6 years ago
Mike Lang f8d10dacdf Audit and fix all usage of dateutil
We wrap direct dateutil calls to handle two distinct cases:

* `common.dateutil.parse()`: We want to handle arbitrary timestamps including tz info,
then convert them to UTC.

This is used in HLS parsing, and for command line input for backfiller

* `common.dateutil.parse_utc_only()`: We want to only handle UTC timestamps,
but datetime.strptime isn't flexible enough (eg. can't handle missing fractional component).

This is used for restreamer request params.
6 years ago
Mike Lang dfc64481a6 Port existing cutting code from restreamer into common
Note this moves over the 'experimental' cutter and deletes the original cutter
that concatenates entire videos before cutting.
We may eventually want to revive that method if the experimental cutter turns out
to introduce too many issues.

We move most of the code over verbatim, but adjust it such that it acts
as a generic iterator that can be used in a variety of contexts.

Some other changes made during the move include telling ffmpeg to be quieter
(don't output version info and junk, only log if something goes wrong),
and avoiding errors during cleanup.
6 years ago
Mike Lang 787b9002ab restreamer: Use correct name for dateutil 6 years ago
Mike Lang 3a1e4b0aef restreamer: Fix missing dependency
This was hidden because common included it
6 years ago
Mike Lang b75b9a9b00 Add stacksampler to all services 6 years ago
MasterGunner a9569d9e96 Removed unneeded '@has_path_args'. 6 years ago
MasterGunner 306ac53d08 Added additional routes for listing available streams and variants. 6 years ago
Mike Lang 901cda4814 Enable backdoor in all services, and add telnet to containers 6 years ago
Mike Lang 9af7795f34 Add gevent.backdoor as an optional arg to all services
Backdoor allows the operator to telnet into the given port, and get a python shell
running inside the process, from which you can debug, modify state (eg. set the log level),
or whatever. This is extremely useful for debugging weird states that you encounter randomly
but can't easily reproduce, without restarting the process and needing to wait until it happens again.
6 years ago
Mike Lang b0ded641c3 Add a logging handler which counts logs for prometheus stats
This isn't as good as having a full centralised logging system, but should
suffice to know if anything funny is happening.
6 years ago
Mike Lang c9d02b3318 restreamer: Prevent prom client blowing up after two different endpoints are hit
Prom client doesn't like you creating two stats with the same name,
even though they have different labels and this makes perfect sense.

I feel like I just need to re-write the prom client at some point - it doesn't actually
do all that much except get in your way, apart from the actual text encoding which I
can steal.

Anyway, in the meantime, we get around this by breaking up metrics into two names,
a "foo_all" and a "foo_ENDPOINT". The foo_all lacks the detailed labels,
but is still labelled by endpoint and can be used more easily.
The foo_ENDPOINT labels have more information but require messier PromQL as you need to
match on a name regex if you want to look at more than one specific endpoint.
6 years ago
Mike Lang 30c4bbec1d restreamer: return the actual response from after_request even if untracked
otherwise any untracked endpoints don't work
6 years ago
Christopher Usher 96e6904c85 Added monotonic to restreamer setup.py 6 years ago
Mike Lang 7525b7c135 restreamer: Add basic prometheus stats to all endpoints
I had to go to some effort to get nice labelling,
which also meant none of the existing libs for this were any good,
but this works well enough.

Exposes the metrics on /metrics.
6 years ago
Mike Lang 17972b87aa Allow setting of log level via WUBLOADER_LOG_LEVEL env var
By using an env var, it is universal and happens prior to arg parsing,
at the same point we do other logging setup.
6 years ago