Paging on a symptom, and dashboarding the cause

Alerts on CPU, memory and disk are alerts on causes, and there are unlimited causes — most of which are entirely compatible with a healthy system.

# pages a human: a user is affected right now
- alert: CheckoutFailing
  expr: |
    sum(rate(http_requests_total{route="checkout",status=~"5.."}[5m]))
    / sum(rate(http_requests_total{route="checkout"}[5m])) > 0.02
  for: 5m
  labels: { severity: page }

# creates a ticket: nobody is affected yet
- alert: DiskWillFill
  expr: predict_linear(node_filesystem_avail_bytes[6h], 4*24*3600) < 0
  for: 1h
  labels: { severity: ticket }

The for clause is the highest-value line in any rule and the one most often left out — an instantaneous threshold fires on every transient spike and transient spikes are constant. Splitting severity into page and ticket is what makes it possible to alert on a disk trend without waking anyone. Everything demoted to a dashboard is still there during an investigation, which is the answer to the objection that pruning loses information.