mirror of https://github.com/ekimekim/wubloader
Fix a deadlock due to signal interactions with prometheus client
The prometheus client uses a threading.Lock() to prevent shared access to certain metric state. This lock is taken as part of doing collection, as well as during metric.labels(). We hit a deadlock where our stack sampler signal arrived during a collection, when the lock was held. This meant that flamegraph.labels() blocked forever, and the lock was never released, hanging all metrics collection. Our solution is a hack, which is to reach into the internals of our metric object and replace its lock with a dummy one. This is reasonably safe, but only as long as the prometheus_client internal structure doesn't change signfigiantly.pull/31/head
parent
c9cc8a73a7
commit
10cca18922
Loading…
Reference in New Issue