One label added in 2025, still the thing that surfaces problems no aggregate shows.
all consumers p95 210ms
integrator-A p95 190ms
integrator-B p95 205ms
integrator-C p95 880ms ← new, since March
integrator-D p95 240ms
C's regression is 4% of traffic and moves the aggregate
by 6ms. the alert that caught it compares each consumer
against the aggregate rather than against a threshold.
A consumer at four times the aggregate is a signal that no absolute threshold catches, because the right threshold differs per endpoint. The cause was an include parameter they had started sending, which was a conversation rather than an index — the second time in a year that has been the answer.