The alert fired at a 500-millisecond p95 because 500 is a round number. Nobody could say where it came from, whether it was too tight or too loose, or what should happen when it fired at four in the afternoon on a busy Tuesday. It had been muted for six weeks.
The symptom
$ grep -c 'alert:' monitoring/rules.yml
41
$ grep -A2 'alert:' monitoring/rules.yml | grep -c 'muted|# disabled'
14
# and the question nobody could answer
# "is 99.9% good?"
# "good compared to what?"
# "...is the site up?"
# "yes"
# "then it is fine"
# meanwhile, that month:
# two incidents, 47 and 12 minutes
# nobody knew whether that was acceptableFourteen of forty-one alerts commented out, and no way to tell whether a fifty-nine-minute month was a good month or a bad one. Both of those are the same problem: there is no agreed target, so every threshold is somebody’s intuition and every incident is argued about individually.
Why it happens
Availability is a business decision presented as a technical one. How much downtime is acceptable is a question about customers, contracts and cost — and it arrives at engineering as “the site should be up”, which is not a target and cannot be measured against.
In the absence of a decision, engineers set thresholds from intuition and then defend them, which is the worst arrangement: nobody outside engineering has agreed to anything, and every alert is negotiable during the incident it fires in.
The fix
The indicator, which is the part that takes a week
SLI the measurement, stated as a ratio
good events / valid events
and both halves are decisions:
good 2xx or 3xx, within 500ms
(not 4xx — a malformed request is not us failing)
(but 429 IS us, because we chose to refuse)
valid requests that count
NOT health checks, NOT the monitoring agent,
NOT known bots, NOT /favicon.ico
the denominator is where the argument is, and it is where
an hour of arguing saves a quarter of meaningless numbers.Excluding health checks is uncontroversial and is routinely forgotten, and on this system it inflated availability by 1.4 points — the monitoring agent hit a trivial endpoint every ten seconds and never failed. The number was flattering and useless.
The 4xx question is the one that produces a genuine argument. A 400 caused by a malformed client request is not the service failing; a 400 caused by validation somebody tightened last week is. Deciding it per endpoint is too fine-grained, and the compromise that held was to exclude 4xx entirely and track them as a separate indicator.
# the recording rule, so the indicator is defined once
- record: sli:checkout_requests:good
expr: |
sum(rate(http_request_duration_seconds_bucket{
route="checkout", status=~"2..|3..", le="0.5"
}[5m]))
- record: sli:checkout_requests:valid
expr: |
sum(rate(http_requests_total{
route="checkout", status!~"4..", user_agent!~"kube-probe|blackbox"
}[5m]))Defining the indicator as a recording rule rather than inline in each alert is what stops three alerts measuring three subtly different things. It also means the definition is in one place when somebody asks what the number covers, which they will.
The objective, and the window
99.5% of valid checkout requests succeed within 500ms,
over a rolling 30 days.
why 99.5 and not 99.9:
99.9% over 30 days = 43 minutes of budget
we had 59 minutes of incidents last month and nobody
complained. 99.9 would have been breached, and the
breach would have meant nothing.
99.5% = 3h 36m. that is a target we can hold and that
constrains us when we exceed it.
an objective nobody would ever act on is set too low.Setting it from what the system already achieves rather than from an aspiration is the part that feels like cheating and is correct: an objective is a promise, and promising something you have never delivered is a way of guaranteeing the number is ignored. Raising it later is a decision with a cost attached, which is the right shape.
The window matters as much as the number. 99.5% over a day and over a quarter are very different promises — the first allows seven minutes and forgets them tomorrow, the second allows ten hours and remembers all of them. A rolling thirty days is the usual compromise and is worth stating explicitly, because everybody assumes a different one.
The error budget, which is what the objective is for
99.5% over 30 days = 3h 36m of budget
spent 47m the payment gateway, on the 4th
12m a bad deploy, on the 11th
3m a certificate renewal, on the 19th
----
62m remaining: 2h 34m (71%)
and the rule, agreed in advance:
budget > 50% ship. deploy on Friday. take risks.
budget < 50% no risky changes. reliability work first.
budget spent feature freeze until the window rolls off.The budget turns reliability from a matter of temperament into arithmetic: “is it safe to ship this on a Friday” has an answer that does not depend on who is asking. That is the whole benefit and it only exists if the rule about exhaustion was agreed by somebody with the authority to enforce a feature freeze.
Getting that agreement is the hard part and it is not a technical conversation. The framing that worked was not “we need better reliability” — it was “we are currently spending this budget without deciding to, and here is what we bought with it”.
Burn-rate alerts, which replace threshold alerts entirely
# 14.4x burn over 1h consumes ~2% of a 30-day budget
- alert: CheckoutBudgetBurningFast
expr: |
(1 - sli:checkout_requests:ratio_rate1h) > 14.4 * 0.005
and
(1 - sli:checkout_requests:ratio_rate5m) > 14.4 * 0.005
for: 2m
labels: { severity: page }
- alert: CheckoutBudgetBurningSlowly
expr: |
(1 - sli:checkout_requests:ratio_rate6h) > 6 * 0.005
and
(1 - sli:checkout_requests:ratio_rate30m) > 6 * 0.005
for: 15m
labels: { severity: ticket }The multiplier is derived rather than chosen: 14.4 over an hour means the alert fires after consuming about 2% of the budget, which is the amount of budget you are willing to lose before being told. That makes the threshold defensible in a way that “500ms” never was.
The short window in each conjunction is doing the resetting rather than the detecting. Without it a one-hour burn rate stays elevated for an hour after the incident is resolved and keeps paging somebody who has already fixed it — which is exactly how the previous fourteen alerts came to be muted.
Two severities from one mechanism is the other benefit: a fast burn is a page and a slow one is a ticket, and both are the same arithmetic with different constants. Forty-one alerts became six.
Latency is a second indicator, not a second number in the first
The temptation is one objective covering availability and speed together, which is what the 500-millisecond threshold in the good-event definition does — and it hides which of the two is failing.
# availability: did it work at all
- record: sli:checkout:availability
expr: |
sum(rate(http_requests_total{route="checkout",status=~"2..|3.."}[5m]))
/ sum(rate(http_requests_total{route="checkout",status!~"4.."}[5m]))
# latency: of the ones that worked, how many were fast enough
- record: sli:checkout:latency
expr: |
sum(rate(http_request_duration_seconds_bucket{
route="checkout", status=~"2..", le="0.5"
}[5m]))
/ sum(rate(http_requests_total{route="checkout",status=~"2.."}[5m]))Two indicators with two objectives and two budgets is more to maintain and it answers a question the combined version cannot: an incident that burns the latency budget and not the availability one is a capacity problem, and the reverse is a correctness problem. Those want different responses and the combined number cannot distinguish them.
The objectives are usually different too. 99.5% availability and 95% latency is a coherent pair — one in twenty requests being slow is tolerable and one in twenty failing is not — and expressing that as a single figure requires picking one of the two and pretending the other does not exist.
Using a histogram bucket boundary as the latency threshold is a practical constraint worth knowing before choosing the number: le="0.5" works because 0.5 is a bucket, and le="0.45" would require interpolating. Choosing the target from what the buckets already are is a small compromise and removes a class of arithmetic error.
The conversation this enables
before:
"can we ship the checkout rewrite this week?"
"...it's risky"
"everything is risky"
(decided by whoever was most confident)
after:
"can we ship the checkout rewrite this week?"
"we're at 71% of budget with 11 days left in the window.
yes. and if it burns 20% we pause and fix it."
the decision is the same shape. it is now made from a number
that somebody outside engineering agreed to.This is the actual deliverable and it is not a monitoring change. The dashboards and the alert rules are the mechanism; the outcome is that a recurring argument has an agreed procedure, and the procedure was agreed when nobody was under pressure.
Verifying it worked
# a quarter later
# alerts: 41 → 6
# muted alerts: 14 → 0
# pages per month: 31 → 4
# of those, actionable: ~6 → 4
# the budget, tracked against the actual incidents
# month 1: spent 62m of 216m (29%)
# month 2: spent 189m of 216m (87%) ← the gateway migration
# month 3: spent 24m of 216m (11%)
# month 2 triggered the rule. feature work paused for nine days.
# nobody argued, because the rule was agreed in January.Month two is the verification that matters: the mechanism produced a decision nobody wanted, and it held because the rule predated the situation. A budget that has never constrained anything is a budget set too generously, and the first time it bites is the test of whether the agreement was real.
Four actionable pages a month from four is the other number — the same real incidents, without the twenty-seven that were not. That is what the burn-rate framing bought, and it is why the muted alerts could be deleted rather than re-enabled.
What this costs
A number that will be argued about forever. The indicator excludes 4xx, and somebody will point out that a validation change breaking every client is invisible to it — which is correct. The objective is 99.5% and somebody will want 99.99% without wanting to pay for it. Both arguments recur and both are healthy; what changed is that they happen in a review meeting rather than during an outage.
The subtler cost is that a single objective per service hides the experience of a minority. A 99.5% overall figure can conceal one customer having a very bad month, one region being consistently slow, or one endpoint being reliably broken — the aggregate is fine and somebody’s experience is not. Adding per-customer or per-region indicators is the answer and multiplies the maintenance, so the honest position is to start with one, know what it hides, and add the second only when somebody has been hurt by the gap.