An hourly spike that made every request four hundred milliseconds slower for two seconds moved the average by nine milliseconds and was invisible in every graph anyone looks at.
// a mean over a minute of requests
// 188ms — and one user in two hundred waited four seconds
// what to record instead: a histogram, then query the quantile
$histogram->observe($durationSeconds, ['route' => $route]);
// p50 tells you the typical experience
// p99 tells you whether the site is embarrassing anyone
// max tells you whether something is periodically brokenAveraging percentiles across servers is arithmetically meaningless, which is why the histogram has to be aggregated rather than the quantile — a mistake that is easy to make and produces numbers that look plausible. Watching the maximum alongside the p99 catches the periodic problem that even a p99 smooths away at low request rates. The p50 and p99 diverging over time is the earliest signal that something is saturating.