Alert on the symptom, never on the cause

An alert on a cause fires when nothing is wrong and stays quiet when something is, because the mapping from causes to consequences is not one to one.

# a cause: fires on a dip that recovered by itself
- alert: PhpFpmProcessesLow
  expr: phpfpm_active_processes < 4

# a symptom: fires when customers are affected
- alert: CheckoutFailing
  expr: |
    sum(rate(http_requests_total{route="checkout",status=~"5.."}[5m]))
    / sum(rate(http_requests_total{route="checkout"}[5m])) > 0.05
  for: 3m

Thirty-one alerts became six by applying this once, and the eleven that fired weekly without ever being actioned were all causes. Causes still belong on a dashboard, where they answer “why” after the symptom alert has fired — the distinction is between what wakes somebody and what they look at once awake. The for clause is what stops a three-second blip paging anybody.