An alert that states a number crossed a threshold hands the receiver a research project, and at three in the morning three sentences beat any precision in the expression.
annotations:
summary: "{{ $value | humanizePercentage }} of checkouts failing"
description: |
Users cannot complete purchases. Check, in order:
1. the payment circuit breaker — redis-cli get cb:payments:open
2. the gateway status page
3. recent deploys
Do NOT restart php-fpm. It loses nothing and destroys the evidence.
runbook: "https://wiki.internal/runbooks/checkout-5xx"
Writing what not to do is as valuable as the checklist, because restarting things is the universal first instinct and it destroys the state that would have explained the incident. Keeping the runbook in the same repository as the rule is what stops the two drifting apart. The first three steps belong inline rather than behind a link — the person reading may be on a phone.