An alert on the error rate fires during any spike and is muted within a week. An alert on how fast the budget is being consumed fires when it matters.
# 14.4x burn over 1 hour = the whole 30-day budget in ~2 days
- alert: CheckoutBudgetBurningFast
expr: |
(1 - (
sum(rate(http_requests_total{route="checkout",status=~"2.."}[1h]))
/ sum(rate(http_requests_total{route="checkout"}[1h]))
)) > 14.4 * 0.005
for: 2m
labels: { severity: page }
The multiplier is chosen from how much of the budget you are willing to lose before being told: 14.4 over an hour means the alert fires after consuming about 2% of a 30-day budget. That framing is what makes the threshold defensible rather than arbitrary. It also fires for a small sustained failure that a percentage threshold would never notice, which is the failure mode that actually costs availability.