From 861fb4de17a8475d9fec12edc4812fe187e32b26 Mon Sep 17 00:00:00 2001 From: Mike Lang Date: Mon, 10 Sep 2018 02:02:48 -0700 Subject: [PATCH] some notes and plans on how new version will work --- spec-notes | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 spec-notes diff --git a/spec-notes b/spec-notes new file mode 100644 index 0000000..b3979b5 --- /dev/null +++ b/spec-notes @@ -0,0 +1,55 @@ + +needs a way to manually set which bots can claim a row + +race condition: row movement + if we try to write to row N, but an insert or delete means the actual data we intended to write + has moved in the interim, we can't easily solve this. But we can easily detect it. + Every row has a unique id that we write. + After writing, we re-read and check id field. + If it's the same, successful write. Note it's still last-write-wins per field. + If it's not, rows were moved! We can't work out what's wrong, + so flag the row for human fixup. + +race condition: claiming videos + In the normal non-hole case, keep it simple. + If you're eligible and current uploader is ineligible or blank, + you write your id to the uploader field. + While processing, you poll the uploader field to check it's still yours. + If you become ineligible, or someone else takes it due to a race, + immediately cancel the work and move on. + Things get trickier if there are holes. + If you are otherwise eligible but have holes, + append a record (row id, your id, missing time) to special sheet, then check if all current bots + have such a record for that row. If not, move on and revisit next poll. + On subsequent checks, no need to re-append a new record. + If all current bots have a record, and yours is the least missing, then either claim it or + flag it depending on setting. + So note that: + - you continue to do meaningful work while you wait for all bots to get back + - if any bot has no holes, it claims as normal and the whole process is abandoned since + the slot is no longer under consideration + +knowing which bots exist: + Entries in a seperate sheet, with a row per bot id, and a ping time. + Ping time should be updated every interval, any bot with a ping time > TTL is dead. + +addressing columns: + you should have an in-code mapping from col name -> position, + positions shouldn't change but just in case. + +thrimbletrimmer sends video trim data directly to wubloader + how? + this is the trigger to go from Awaiting Edits to Processing Edits + +chunk sheet special cases: + no draft step, go straight from queued to processing video at full res to uploading + upload unlisted + +dealing with holes + if only some bots (or none) have holes: + bots without holes should take it + if all bots have holes: + if it's chunk sheet: + upload from one with least holes + if it's main sheet: + flag for human intervention, allow for force override