Measure the p99 and the max; the average hides incidents

An hourly spike that made every request four hundred milliseconds slower for two seconds moved the average by nine milliseconds and was invisible in every graph anyone looks at.

// a mean over a minute: 188ms
// and one user in two hundred waited four seconds

// record a histogram, query the quantile
$histogram->observe($durationSeconds, ['route' => $route]);

// p50 — the typical experience
// p99 — whether the site is embarrassing anyone
// max — whether something is periodically broken

Averaging percentiles across servers is arithmetically meaningless, which is why the histogram has to be aggregated rather than the quantile — a mistake that is easy to make and produces numbers that look plausible. Watching the maximum alongside the p99 catches the periodic problem that even a p99 smooths away at low request rates. The p50 and p99 diverging over time is the earliest available signal that something is saturating.