An alert with a runbook link, or no alert

An alert that fires at three in the morning and does not say what to do is a notification that somebody else has a problem.

- alert: QueueTimeToStartHigh
  expr: queue_oldest_pending_seconds{queue="interactive"} > 60
  for: 5m
  annotations:
    summary: "interactive queue: oldest job {{ $value }}s old"
    runbook: "https://wiki.internal/runbooks/queue-backlog"
    dashboard: "https://grafana.internal/d/queues"

Requiring a runbook link before an alert may be created is a policy that removes about a third of proposed alerts on its own — writing the runbook forces the question of what the responder would actually do, and often the answer is nothing. The two rules that could not get one were both “investigate”, which is not a procedure and became dashboard panels instead.