Commit Graph

83 Commits (91cc3311281943eb4de5c9e92db2c30c1b5f1ec9)

Author SHA1 Message Date
Mike Lang 101468cbca Fix a bug when reporting errors claiming candidates
The SQL args were the wrong way around, so we tried to set error to the id.
Thankfully this failed as the id queried is not a valid uuid.

Fix by making args named like every other usage of query()
3 years ago
Mike Lang 5e60344522 cutter: Fix errors due to unsupported cut type not being handled correctly
Previously, determining the correct cut type and setting up the cut iterator
wasn't an operation we expected to be able to fail.
It happened outside all error handling blocks.

However, due to the multi-range work, this can now fail if we are requesting
an unsupported combination of transitions and cut type.

In order to correctly handle this like other cut errors, ie. set to UNEDITED
and set error column, we move this logic into the upload_wrapper right before we iterate
through the resulting cut.
3 years ago
Mike Lang a47c29fff4 Link images to github repo by adding a LABEL
When pushed, this tells github to associate the ghcr.io repo that was pushed to
with the github repo specified (the owner needs to match).

This does a few things.
Most importantly, this automatically gives github actions credentials to push to these
repositories when run in the context of the wubloader repo.
3 years ago
Mike Lang aab8cf2f0f Set up plumbing for multi-range videos and implement no-transition fast cut videos only
This is the simplest case as we can just cut each range like we already do,
then concat the results.

We still allow for the full design in the database and cutter, but error out if transitions
is ever anything but hard cuts or if it's a full cut.

We also update the restreamer to allow accepting ranges, however for usability we still allow
the old "just one start and end" args.

Note this changes the thrimshim API to give and take the new "video_ranges" and "video_transitions" columns.
3 years ago
Mike Lang 62bd6539ea Unpin gevent as that was a workaround for a py2 issue 3 years ago
Mike Lang 21856c68aa Fix all instances of file.write() for py3
In python 3, file.write() may do a partial write and returns the number of characters written.
In order to not lose data, we need to wrap every instance of file.write() with our new
common.writeall() wrapper that loops until the data is actually written.
3 years ago
Mike Lang a56f6859bb more py3 fixes 3 years ago
Mike Lang f2a8007bf7 Fix build dependency issues 3 years ago
Mike Lang 50231a5154 py3 fixes for cutter 3 years ago
HubbeKing 6d790a1b36 Do a first naive pass for py3 compatibility
Check that open() calls for reading and writing use binary modes
Use alpine version with py3-pip package
Use python3 in Dockerfile CMD
Remove sys.setdefaultencoding() "hack"
Simplify ensure_directory() in common.common package
3 years ago
Mike Lang f0546e2ee3 Pin gevent to 1.5a2 to avoid https://github.com/gevent/gevent/issues/1711 3 years ago
Mike Lang b53fcd65a0 Add dependencies required to install psycopg2 from source
We can't install the binaries as they don't support musl
4 years ago
HubbeKing 86f7823348 Replace calls to gevent.signal() with gevent.signal_handler()
gevent.signal() was removed in gevent 1.5a4, see http://www.gevent.org/api/gevent.signal.html
Removed on Feb 5th, see https://github.com/gevent/gevent/pull/1530
4 years ago
Mike Lang a53786dc2d Add file and make as build dependencies
gevent now requires these to build. I'm not sure when this changed.
4 years ago
Mike Lang b9cd76b1a2 Add non-static implict tags in sheetsync
In order for the upcoming playlist manager to be able to use the DB `tags` column to know
what tags a video has, all the tags it needs need to be present.

Previously, this was a problem because the day and category tags only get added at the cutter
and so wouldn't be listed.

This moves them so they are added when parsing the row in sheetsync.
It also adds the poster moment tag if poster moment is checked.

Note that fully static tags that go on all videos are still only added in cutter,
but the playlist manager doesn't need to care about those (since by definition
they will match every video).
4 years ago
Mike Lang a30d595fee cutter: Use video's custom tags when uploading 4 years ago
Mike Lang 4a2fe7a6ed cutter: Explicitly set mime type of uploads correctly 5 years ago
Mike Lang fde2758275 cutter: Fix bug where uploader was cleared on non-retryable error
Instead of on retryable error
5 years ago
Mike Lang a002619c4c youtube upload: Explicitly set mime type 5 years ago
Mike Lang a438d86f80 cutter: Fix multiple problems with logging errors 5 years ago
Mike Lang 4bbcc8bc06 Revert "Merge pull request #155 from ekimekim/mike/manual-uploads"
This reverts commit 99de586353, reversing
changes made to 4b04f70b6f.

We don't need this feature and it complicates things and adds bugs.
5 years ago
Mike Lang c580671da2 Create upload pending state 5 years ago
Mike Lang 605fedeb52 cutter: Log when upload errors occur
This got lost in the shuffle at some point.
5 years ago
Mike Lang e4706232ad cutter: Fix bug in handling cancellation before finalize
it would always think it's finalized
5 years ago
Mike Lang 4c0e20983c Local upload backend: Always use .ts extension
Now that we're doing that for both fast and full cuts.
5 years ago
Mike Lang 1cb4d41334 cutter: Set upload time when marking video as done without transcoding 5 years ago
Mike Lang a7ac264201 cutter: Fix local upload backend, which was returning video id as a uuid
instead of a string
5 years ago
Mike Lang da58021c6f cutter: Fix typo where encoding settings had wrong name 5 years ago
Mike Lang 1ef15780df cutter: Fix lack of info in error handling 5 years ago
Mike Lang c1cb7966af cutter: Default to full cuts, not fast 5 years ago
Mike Lang b2a07ef114
Merge pull request #140 from ekimekim/mike/build-improvements
Refactor dockerfiles for more shared layers
5 years ago
Mike Lang 731ef9e2d0 Refactor dockerfiles for more shared layers
By carefully ensuring most of our dockerfiles are identical in their first few layers,
we only need to build those layers once instead of every time.

In particular, we move installing gevent to before installing common,
so that even when common changes gevent doesn't need to be reinstalled.

This is important because gevent takes ages to install.

Also fixes segment_coverage, which wasn't being installed.
5 years ago
Mike Lang 3dfe0e8722 Allow thrimshim to safely cancel a job while it is cutting
This differs from the existing reset row by only suceeding if the upload is not
in finalizing.

We also make some changes to cutter to handle this situation gracefully.
5 years ago
Mike Lang 4d52b18b04 cutter,restreamer: Set stream=True for full cuts when appropriate
And also default to a new ffmpeg encoding setting for high-quality mpegts
(ie. still streamable) that is encoded very quickly.
5 years ago
Mike Lang f9b48bc70e cutter: Add more metrics 5 years ago
Mike Lang 52e6c4ad41 sheetsync, cutter: Collect metrics on http calls
In particular, to google apis.
5 years ago
Mike Lang 17af1c4e89 cutter, sheetsync: Wait for DB to connect on startup
This is a nicer error than crashing in the depths of some error handler
(which is what will happen if the DB goes unavailable while they're running),
and it's a far more common case (eg. the DB is misconfigured) than having it fail
halfway through.

Neither of these services can do anything meaningful without the DB,
so crashing without it is acceptable behaviour.
5 years ago
Mike Lang ba746ff6e6 Add title and description header/footer in thrimshim instead of cutter
This accomplishes two things:
1. It allows thrimshim to properly validate length restrictions (not implemented yet)
2. It means that the database has a record of the values actually written for each of these rows,
instead of that information depending on how the cutter was configured at the time.
5 years ago
Mike Lang 596cd92644 cutter: Add more specific error handling to upload backends
For youtube, know that 4xx's are safe even if finalizing was set.

For local, make all disk errors retryable since it doesn't matter.
5 years ago
Mike Lang 736040435c Refactor error handling in uploads
Instead of handling each error condition seperately,
we raise an UploadError which includes whether it's retryable.

The advantage of this is that upload backends can also raise an UploadError
to indicate two conditions it currently cannot:
	That an error is unretryable
	That an error is retryable, even if the row was already in finalizing

Under this scheme, errors while cutting become unretryable UploadErrors,
and unhandled exceptions in uploading become retryable UploadErrors if
the row is not yet finalizing only.
5 years ago
Mike Lang 40458d9d7f local backend: Use original version of title in write_info
instead of safe version
5 years ago
Mike Lang 3fbbe59b00 cutter Local backend: Fix typo and file extension when full cutting 5 years ago
Mike Lang b0a71fd9f1 cutter: Fix mistake in check_candidate error handling 5 years ago
Mike Lang ba53172cbc cutter: Get video link from backend instead of hard-coding youtube 5 years ago
Mike Lang ed2395d08e cutter: Fix typo that prevented backends from being configured 5 years ago
Mike Lang 1218b694dd cutter: Allow fast or full cuts per location
On full cut, use encoding settings determined by the backend.
5 years ago
Mike Lang d3e1d6b4fc Resurrect non-experimental cut, now dubbed "full" (vs "fast") cut
In a fast cut, we edit the first and last segments then concatenate them all.
However, this leads to some tiny but perciptible artifacting around the border
of the first and second (and second-last and last) segments.

A full cut is much slower, but re-encodes the video into the desired format
and is more reliable.

We want both options to be available.

With this commit, we only add the option, we don't use it in restreamer or cutter.
5 years ago
Mike Lang 40c4baef0f youtube upload: Set category and language settings
configured on a per-location basis.
5 years ago
Mike Lang 5e0d5b9ddc cutter: Add category and sheet_name as video tags
This gives us tags on categories, and on which day's sheet the event came from.

We use these to make automatic playlists.
5 years ago
Mike Lang cbd0ef3d9e cutter: Add title header, description footer and static tags
These are pre-canned parts of the video metadata that we want to be configurable.
5 years ago