Alert on symptoms; put the causes on a dashboard

CPU, memory and disk are causes, and there are unlimited causes — most of them entirely compatible with a healthy system serving every request correctly.

# pages a human: a user is affected right now
- alert: CheckoutFailing
  expr: |
    sum(rate(http_requests_total{route="checkout",status=~"5.."}[5m]))
    / sum(rate(http_requests_total{route="checkout"}[5m])) > 0.02
  for: 5m
  labels: { severity: page }

# creates a ticket: nobody is affected yet
- alert: DiskWillFill
  expr: predict_linear(node_filesystem_avail_bytes[6h], 4*24*3600) < 0
  for: 1h
  labels: { severity: ticket }

The test for whether something should wake a person is whether a customer could tell. Everything failing that test goes on a dashboard, where it is available during an investigation and silent the rest of the time. Splitting severity into page and ticket is what makes it possible to alert on a disk trend without waking anyone, and without that split the only options are to page for everything or to alert on nothing.