Percentiles labelled by consumer, and the outlier

One label on a histogram, and a p95 that had been describing nobody.

GET /api/v2/orders, one week:

  all              p50 42ms   p95 220ms

  integrator-a     p50 40ms   p95 190ms
  integrator-b     p50 44ms   p95 210ms
  integrator-c     p50 41ms   p95 205ms
  integrator-d     p50 1,880ms  p95 6,400ms

d is 2% of requests. the aggregate p95 barely moves.

An aggregate percentile is a weighted average of populations that may have nothing in common, and a small consumer with a pathological pattern is invisible in it. One label costs nothing until the cardinality is unbounded — four named integrators is safe and the same label on user id would not be.