Alert on symptoms, dashboard the causes

Alerting on every cause — disk, memory, queue depth, replica lag, a failed cron — produces an alert per cause and a channel nobody reads. Alerting on what a user would notice produces a handful, each of which is worth waking for.

# symptom: alert
- error rate > 2% of requests for 5m
- p95 latency > 2s for 10m
- checkout success rate < 98% for 5m

# cause: dashboard, and a link from the alert
- buffer pool hit rate, queue depth, replica lag, disk

The causes still matter — they are how the alert gets diagnosed — and they belong on the dashboard the alert links to rather than in the pager. The test for whether something should alert: if it fires at three in the morning and the answer is “look at it tomorrow”, it was never an alert.