An alert saying HighErrorRate on shop-web at three in the morning is a puzzle. The person receiving it may not have written the check, and the context lives in someone’s head.
- alert: CheckoutSuccessRateLow
for: 5m
annotations:
summary: 'Checkout success below 98% ({{ $value | printf "%.1f" }}%)'
impact: 'Customers cannot complete orders.'
runbook: 'https://wiki.internal/runbooks/checkout-success-rate'
dashboard: 'https://grafana.internal/d/checkout'The runbook does not have to be good — three bullet points on what to check first beats nothing by a wide margin, and it improves after each incident. What it must be is linked from the alert rather than findable in principle. Adding the current value to the summary saves a lookup and often answers “is this bad” on its own.