Change to backfiller.main to backfill the last 3 hours on start up before doing a full backfill

pull/24/head
Christopher Usher 6 years ago
parent 176633bf7d
commit 48d11045d4

@ -233,7 +233,7 @@ def backfill_node(base_dir, node, stream, variants, hours=None, start=None,
def main(base_dir, stream, variants, fill_wait=5, full_fill_wait=180, sleep_time=1):
"""Prototype backfiller service.
Do a full backfill of stream/variants from all nodes initially. Then every sleep_time minutes check to see if more than fill_wait minutes have passed since the last backfill. If so do a backfill of the last 3 hours. Also check whether it has been more than full_fill_wait minutes since the last full backfill; if so, do a full backfill."""
Do a backfill of the last 3 hours from stream/variants from all nodes initially before doing a full backfill from all nodes. Then every sleep_time minutes check to see if more than fill_wait minutes have passed since the last backfill. If so do a backfill of the last 3 hours. Also check whether it has been more than full_fill_wait minutes since the last full backfill; if so, do a full backfill."""
# TODO replace this with a more robust event based service and backfill from multiple nodes in parallel
# stretch goal: provide an interface to trigger backfills manually
# stretch goal: use the backfiller to monitor the restreamer
@ -241,6 +241,7 @@ def main(base_dir, stream, variants, fill_wait=5, full_fill_wait=180, sleep_time
fill_start = datetime.datetime.now()
full_fill_start = fill_start
backfill(base_dir, stream, variants)
backfill(base_dir, stream, variants, order='random')
# I'm sure there is a module that does this in a more robust way

Loading…
Cancel
Save