Commit Graph

94 Commits (649a0fb2ddb141a5412fc4afa8acd3f2ebd32cdc)

Author SHA1 Message Date
Mike Lang c8724a1e63 rewrite fast cuts to support transitions being allowed later
In theory there should be no change in actual output for no-transition cuts,
even though we're handling the logic in a very different way.

This doesn't actually allow transitions, but sets up most of what is needed
4 months ago
Mike Lang 066d10f94a Full cut: Support video transitions
We support all preset transitions in the xfade filter,
as well as a handful of "custom" ones we define.

We only support an audio cross-fade. We may want to support J and L audio cuts (switch audio
before/after the transition) later.
4 months ago
Mike Lang 5fbdaf8422 full cuts: Support multiple ranges
This allows full cuts to support multiple ranges in the same way fast cuts do,
by using multiple inputs to ffmpeg and concat filters joining them.

This will be easy to add transitions to later as this is "just" replacing a concat filter
with an xfade + afade filter.
4 months ago
Mike Lang f9ff537b84 restreamer: Fix a bug where playlist reports video is finished spuriously
This happens when we are live viewing a stream, and the last available segment
is at the end of an hour.
We generate the end timestamp as being the end of the last available hour,
which might be within the range of the last available segment. When this happens
we stream the last segment then say we reached the requested end point.
This makes the player stop asking for more segments.

The fix is to pad the end time by an extra hour so we're asking for 1 hour more than the
last available hour.
7 months ago
Mike Lang daf6001402 restreamer: Move get_best_segments() inside playlist cache 7 months ago
Mike Lang 0895ce44ff generate media playlist: Yield in chunks for performance
Flask sends a chunked response with one chunk per item yielded.
This adds a lot of overhead per yielded item.
We avoid this by collecting the lines of the media playlist into larger chunks
and only flushing once every 1000 segments.

For small playlists this means they'll be emitted as one chunk,
but for large playlists we still get the streaming behaviour.
7 months ago
Mike Lang 1857a998c9 reduce overhead of gevent.idle() by only yielding once per 1000 segments 7 months ago
Mike Lang 7e90346b87 restreamer: Have playlist cache stream results 7 months ago
Mike Lang 909cece196 restreamer: Move media playlist cache to also cover HLS playlist generation 7 months ago
Mike Lang cb2c58c75c restreamer: Yield the event loop during playlist generation 7 months ago
Mike Lang be7a9cd1fa restreamer: Make media playlist streamed
To deal better with very large playlists
7 months ago
Mike Lang a6d4f2466a restreamer: coalesce concurrent requests for the same playlist URL 7 months ago
Mike Lang 3606fadaa8 Pin gevent version to work around build issues
Seeing the following error on latest versions of gevent:

 Traceback (most recent call last):
   File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
     return _run_code(code, main_globals, None,
   File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
     exec(code, run_globals)
   File "/usr/lib/python3.9/site-packages/zulip_bots/schedulebot.py", line 2, in <module>
     import gevent.monkey
   File "/usr/lib/python3.9/site-packages/gevent/__init__.py", line 72, in <module>
     from gevent._hub_local import get_hub
   File "/usr/lib/python3.9/site-packages/gevent/_hub_local.py", line 150, in <module>
     import_c_accel(globals(), 'gevent.__hub_local')
   File "/usr/lib/python3.9/site-packages/gevent/_util.py", line 148, in import_c_accel
     mod = importlib.import_module(cname)
   File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'gevent._gevent_c_hub_local'
12 months ago
Mike Lang e689626815 Add a small time range around the timestamp when extracting a frame
This should hopefully result in frames on the edge of timestamps being extracted
from a combination of the neighboring segment and the naive one,
so that we don't get errors extracting a frame.
1 year ago
Mike Lang 78c053000e Upgrade pip in order to make wheels work 1 year ago
Mike Lang 044dfb8084 Pin argh to avoid stupid breaking changes 1 year ago
Mike Lang 4fb039d7d9 restreamer: Expose an endpoint for listing files that aren't segments 1 year ago
Mike Lang 9df51e80fb add smart cut to restreamer 1 year ago
Mike Lang 30d5ccc483 Fix all old references to github.com/ekimekim/wubloader 1 year ago
ElementalAlchemist e13333d2b4 Fix BANDWIDTH specification in playlist 2 years ago
Mike Lang a6797aed19
Make our HLS master playlist spec compliant by adding bandwidth (#323)
Even though it's a complete guess.
2 years ago
Mike Lang c493869b9a Have list_segment_files also list chat archives
Otherwise backfilling of chat doesn't work
2 years ago
Mike Lang 36be1bc01a restreamer: Don't 500 on bad timestamps in chat endpoint 2 years ago
Mike Lang 834edf2701
Don't show a template option for non-png files (#312)
Each template now has two files, a `.png` and a `.json`. This is currently making them show up twice.
To fix this, we only consider files which end in `.png`.

We do this in the backend so the frontend doesn't need to know about it.
2 years ago
Mike Lang 31c3f0b8b1 OS-level dependencies for Pillow 2 years ago
Mike Lang 0f0aee36b3 restreamer: Add endpoint for previewing thumbnails 2 years ago
Mike Lang 08257386e2 Add restreamer endpoint for viewing chat messages 2 years ago
Mike Lang f8b3ace148 Backfill chat archives under the "chat" quality 2 years ago
Mike Lang 1add3c5c22 Implement tombstoning to allow for segment deletion
Rarely, we find ourselves needing to explicitly delete some data, eg. something that shouldn't
have been public and should be removed from all records.

It would also be nice if we could "clean up" bad versions of the same segment,
which occasionally come up when downloaders have issues.

With our distributed segment database, this is actually rather difficult as deleting the data
from any one server would cause it to be restored from the others. It was only possible
by stopping all backfill, deleting the data on all servers, then starting backfill again.

Here we introduce a more practical approach. An operator creates an empty flag file
with the same name as the segment to be deleted, but with a `.tombstone` extension.
eg. to delete a file `/segments/desertbus/source/2019-11-13T02/45:51.608000-2.0-full-7IS92rssMzoSBQDIevHStbTNy-URRV3Vw-jzZ6pwOZM.ts`,
you would create a tombstone `/segments/desertbus/source/2019-11-13T02/45:51.608000-2.0-full-7IS92rssMzoSBQDIevHStbTNy-URRV3Vw-jzZ6pwOZM.tombstone`.

These tombstone files do two important things:
* They hide the segment from being listed, which both means:
  * It can't be restreamed or put into a video
  * It can't be backfilled to other nodes
* The tombstone files themselves do get backfilled to other nodes, so you only need to mark them on one server.

Once the tombstone has propagated to all nodes, the segment file can be deleted independently on each one.

We chose not to have a tombstone automatically trigger a segment deletion for safety reasons.
2 years ago
Mike Lang 871925aef5 restreamer: Disallow implicit time ranges over 12h
Fixes #286
3 years ago
Mike Lang 9f9ef66a85 Add endpoint to get a given frame of video 3 years ago
Mike Lang ac44298299 restreamer: Disallow very long waveform images
These can take a LOT of server resources (RAM, IO), so we'd rather just fail out
on very long edits where the resulting image would be useless anyway.
3 years ago
ElementalAlchemist 56699d5737 Put quality name in the place where clients read it (or at least HLS.js does) 3 years ago
Mike Lang 7649a4e840 Improve WSGIServer graceful shutdown handling
Previously both restreamer and thrimshim had some complex logic for dealing with
graceful shutdown, in different ways, that was still prone to race conditions.

We replace this with a common method that does it properly.

Fixes #226
3 years ago
Mike Lang a47c29fff4 Link images to github repo by adding a LABEL
When pushed, this tells github to associate the ghcr.io repo that was pushed to
with the github repo specified (the owner needs to match).

This does a few things.
Most importantly, this automatically gives github actions credentials to push to these
repositories when run in the context of the wubloader repo.
3 years ago
Mike Lang aab8cf2f0f Set up plumbing for multi-range videos and implement no-transition fast cut videos only
This is the simplest case as we can just cut each range like we already do,
then concat the results.

We still allow for the full design in the database and cutter, but error out if transitions
is ever anything but hard cuts or if it's a full cut.

We also update the restreamer to allow accepting ranges, however for usability we still allow
the old "just one start and end" args.

Note this changes the thrimshim API to give and take the new "video_ranges" and "video_transitions" columns.
3 years ago
Mike Lang 3de44d6731 Add ability to render waveforms in restreamer 3 years ago
Mike Lang 62bd6539ea Unpin gevent as that was a workaround for a py2 issue 3 years ago
Mike Lang f2a8007bf7 Fix build dependency issues 3 years ago
Mike Lang 9fad66c6be py3 fixes for restreamer 3 years ago
Mike Lang d03ae49eec Remove defunct "smart cut" method
This was an alternate way of doing a cut that turned out to work exactly the same as a fast cut,
just with a more complex implementation.
3 years ago
HubbeKing 6d790a1b36 Do a first naive pass for py3 compatibility
Check that open() calls for reading and writing use binary modes
Use alpine version with py3-pip package
Use python3 in Dockerfile CMD
Remove sys.setdefaultencoding() "hack"
Simplify ensure_directory() in common.common package
3 years ago
Mike Lang f0546e2ee3 Pin gevent to 1.5a2 to avoid https://github.com/gevent/gevent/issues/1711 3 years ago
Mike Lang fe4299e926 Change the generate_videos endpoint to make mkvs according to an hours spec passed in 3 years ago
Mike Lang efe185bc0c Fix copy-paste error in restreamer metrics endpoint 4 years ago
Mike Lang 48ef416dfb restreamer, thrimshim: Allow /metrics/* in addition to /metrics
This allows the metrics proxying from nginx to work even if the path is not rewritten,
which is hard to do in k8s.
4 years ago
HubbeKing 86f7823348 Replace calls to gevent.signal() with gevent.signal_handler()
gevent.signal() was removed in gevent 1.5a4, see http://www.gevent.org/api/gevent.signal.html
Removed on Feb 5th, see https://github.com/gevent/gevent/pull/1530
4 years ago
Mike Lang a53786dc2d Add file and make as build dependencies
gevent now requires these to build. I'm not sure when this changed.
4 years ago
Mike Lang 4d21f447ad Add restreamer call to generate full concat'd videos
This makes the timelapse easier.
5 years ago
Mike Lang b39e844c1e restreamer: Fix missing import of smart cut 5 years ago