segment_coverage: Fix a problem where metrics would fail

Because the checking process is entirely CPU-bound, it does not give any other
greenlets a chance to run while it is processing. This prevents us from responding
to metrics queries, and prometheus then times out.

By stopping to handle all other traffic in between each hour processed, we ensure metrics
remain responsive while processing.
pull/200/head
Mike Lang 4 years ago committed by Mike Lang
parent 13a228070a
commit 15c357509f

@ -261,6 +261,11 @@ class CoverageChecker(object):
all_hour_holes = {} all_hour_holes = {}
all_hour_partials = {} all_hour_partials = {}
for hour in hours: for hour in hours:
# Let other things run, to avoid starving them with CPU-heavy workload
# (in particular the metrics server can have issues responding in time
# otherwise).
gevent.idle()
if self.stopping.is_set(): if self.stopping.is_set():
break break
self.logger.info('Checking {}/{}'.format(quality, hour)) self.logger.info('Checking {}/{}'.format(quality, hour))

Loading…
Cancel
Save