Wubloader is a system for saving, re-serving and cutting into videos of a target
twitch (or probably other HLS, but some twitch specifics are assumed) stream.

It was designed to serve the needs of the [Video Strike Team](https://vst.ninja)
as part of [Desert Bus For Hope](https://desertbus.org).

A full design doc can be read at [initial-design-doc.pdf](./initial-design-doc.pdf),
but a brief overview of the components:

#### Shared Components

* `common` provides code shared between the other components.

#### Ingest
* `downloader` grabs segments from twitch and saves them to disk
* `restreamer` serves segments from disk as well as playlist files allowing them to be streamed
* `backfiller` queries restreamers of other servers in order to pick up segments this server doesn't have already,
  ie. it replicates missing segments.
* `chat_archiver` records twitch chat messages and merges them with records from other nodes.

#### Processing
* `sheetsync` syncs specifc database columns to a google doc which is the primary operator interface.
* `cutter` interacts with a database to perform cutting jobs
* `thrimshim` acts as an interface between the `thrimbletrimmer` editor and the database.
* `thrimbletrimmer` is a browser based video editor.
* `segment_coverage` regularly checks whether there is complete segment coverage for each hour. 
* `playlist_manager` adds videos to youtube playlists depending on tags.

#### Analysis
* `bus_analyzer` does OCR on the stream to produce timestamps of/for progress through the route.
* `buscribe` is back-end audio speech-to-text extraction.
* `buscribe_api` is the API provided for below to access the stored results of transcription.
* `buscribe-web` is the web frontend to interface with `buscribe`'s output (via `buscribe_api`)

#### Services
* `postgres` hosts a Postgres database to store events to be edited.
* `nginx` provides a webserver through which the other components are exposed to the outside world.
* `monitoring` provides dashboards to allow the wubloader to be monitored.

### Usage

All components are built as docker images.
Components which access the disk expect a shared directory mounted at `/mnt`.

A docker-compose file is provided to run all components. See `docker-compose.jsonnet`
to set configuration options, then generate the compose file with `./generate-docker-compose`.
Then run `docker-compose up`.

There is also a kubernetes-based option, but it is less configurable and only fully supports replication and editing nodes.
Basic support for running the database and playlist_manager has been added, but not tested.
See [k8s.jsonnet](./k8s.jsonnet) for details.

Further details of installing and configuring the backfiller are provided in [INSTALL.md](./INSTALL.md).