The reliability target had been written into a document in 2020, quoted in two client contracts, and never measured. When an incident review asked whether the quarter had met it, three people produced three numbers — 99.94%, 99.7% and 98.2% — all correct, all measuring different things.
The symptom
"did we meet 99.9% last quarter?"
ops: 99.94% — uptime of the load balancer health
check, which returns 200 whenever
nginx is running
backend: 99.7% — non-5xx responses across all routes,
including an internal admin endpoint
that 500s constantly and nobody uses
support: 98.2% — hours during which at least one
customer reported a problem
three numbers. no way to reconcile them. and a contract
that says 99.9% without saying of what.None of the three is wrong and none of them is the number the contract meant. The health check answers whether a process is running, the aggregate error rate answers a question nobody asked, and the support figure measures reporting rather than availability.
Why it happens
A percentage is easy to write into a document and requires four decisions to be measurable: what counts as an event, which events are valid, which of those are good, and over what window. None of those were made in 2020 because the number felt self-explanatory.
The fix
Choosing the indicator, which is a decision about the denominator
# good events
sum(rate(http_requests_total{
route="checkout", status=~"2..|3.."
}[5m]))
/
# VALID events — 4xx excluded, deliberately
sum(rate(http_requests_total{
route="checkout", status!~"4.."
}[5m]))what excluding 4xx means, stated so it can be argued with:
a client sending a malformed request is not our
availability problem
and: a bug that returns 400 for a VALID request looks
like perfect availability
the alternative — counting everything — makes a scripted
attack look like an outage.
we excluded them, and added a separate alert on the 4xx
RATE so the blind spot has its own signal.The blind spot created by the exclusion is real and the mitigation is a second alert rather than a different denominator, because a denominator that includes client errors is dominated by them during any scan. Writing down which was chosen and why is what makes the number defensible six months later.
Scoping it to checkout rather than to all routes is the other decision. An aggregate across every endpoint is dominated by whatever is highest-volume, which on this system was an internal polling endpoint — so the headline number moved when the polling interval changed, which is nonsense.
Latency as a second indicator, not a second clause
# availability: did it work at all
- record: sli:checkout:availability
expr: |
sum(rate(http_requests_total{route="checkout",status=~"2..|3.."}[5m]))
/ sum(rate(http_requests_total{route="checkout",status!~"4.."}[5m]))
# latency: of the ones that worked, how many were fast enough
- record: sli:checkout:latency
expr: |
sum(rate(http_request_duration_seconds_bucket{
route="checkout", status=~"2..", le="1.0"
}[5m]))
/ sum(rate(http_requests_total{route="checkout",status=~"2.."}[5m]))Two indicators with two objectives answer a question the combined version cannot: an incident burning the latency budget and not the availability one is a capacity problem, and the reverse is a correctness problem. Those want different responses and a single number cannot distinguish them.
The threshold has to be a histogram bucket boundary — le="1.0" works because 1.0 is a bucket and le="0.85" would require interpolating. Choosing the objective from the buckets that already exist is a small compromise that removes a class of arithmetic error, and re-bucketing later loses the history.
The objective as a purchase
the conversation, with the person who owns the product
rather than within engineering:
"at 99.9% you get 43 minutes of failure a month.
at 99.99% you get 4, and it costs a second region,
a redesign of the payment path, and a quarter of
somebody's time.
which do you want to buy?"
the answers:
checkout 99.9% availability
95% under 1s
product pages 99.5%
the admin panel 99% ← nobody would have guessedPresenting it as a purchase is what makes the conversation productive, because the trade is genuinely between reliability and everything else that quarter. The admin panel at 99% is the answer that would never have come from engineering and is obviously correct once stated — nobody loses money when an internal tool is down for six hours a month.
It also resolved the contract question. The number quoted to clients was for checkout, which is what they care about, and the document was amended to say so — which took a lawyer twenty minutes and had been an unstated ambiguity for two years.
The error budget, and spending it deliberately
99.9% over 30 days = 43m 12s of budget.
Q2, as it was actually spent:
a planned migration cutover 18m 20s
one incident (payment gateway) 9m 04s
a deploy that was rolled back 4m 11s
background noise 2m 40s
----------------------------------------
spent 34m 15s
remaining 8m 57s
and the policy, agreed BEFORE it was first needed:
budget exhausted → no non-essential deploys until the
window resets. reliability work takes priority.Spending eighteen minutes on a planned migration is the healthiest possible use of a budget — a scheduled, understood risk taken because the budget existed to cover it. Before this existed the same cutover would have been done at three in the morning to avoid being noticed, which is worse in every respect.
The policy has to be agreed in advance rather than negotiated during, because the first time a budget is exhausted is the first time somebody has a release they consider essential. Writing it down while nothing is on fire is the only moment it can be written honestly.
Burn-rate alerts, and retiring what they replace
# fast: 14.4x burn over 1h — a 30-day budget in ~2 days
- alert: CheckoutBudgetBurningFast
expr: |
(1 - sli:checkout:availability_1h) > 14.4 * 0.001
and (1 - sli:checkout:availability_5m) > 14.4 * 0.001
for: 2m
labels: { severity: page }
annotations:
runbook: 'https://wiki/runbooks/checkout-availability'
# slow: 3x over 6h — noticeable, not urgent
- alert: CheckoutBudgetBurningSlowly
expr: |
(1 - sli:checkout:availability_6h) > 3 * 0.001
and (1 - sli:checkout:availability_30m) > 3 * 0.001
labels: { severity: ticket }The short second window in each expression is what makes the alert stop firing promptly once the condition clears — without it, a four-minute incident produces an alert that stays open for an hour and the third time that happens somebody adds a silence rule.
and the alerts these replaced, which were deleted:
HighErrorRate any 5xx above 1%, any route
SlowResponses p95 above 2s, any route
PhpFpmProcessesLow a cause, not a symptom
DatabaseConnectionsHigh a cause
DiskSpaceLow kept — it is not an SLI, and it
is genuinely actionable
11 alerts → 4. and the three that were deleted had fired
41 times in the quarter with no action taken on any.The monthly review
one hour, monthly, with the product owner present:
budget spent, and on what
whether the objective is still the right number
the actions from the last review, and whether they
happened
what came out of the first four:
month 1 the objective was too tight for product pages.
lowered from 99.9% to 99.5%.
month 2 the checkout latency objective was measuring
the wrong bucket. corrected.
month 3 no change. the budget was 40% spent.
month 4 budget exhausted on day 19. the policy was
invoked for the first time, and held.Lowering an objective in month one is not a failure — it is the process working. A number nobody can meet is a number everybody learns to ignore, and adjusting it once with the product owner in the room is what makes the remaining ones credible.
Verifying it worked
$ curl -sG http://prometheus:9090/api/v1/query
--data-urlencode 'query=avg_over_time(sli:checkout:availability[30d])'
| jq -r '.data.result[0].value[1]'
0.99921
# one number, one definition, and the query is in the
# repository next to the alert rules.
$ ./bin/slo-report --quarter=Q2
checkout availability 99.921% target 99.9% ✓
checkout latency p95 96.2% target 95% ✓
product pages 99.61% target 99.5% ✓
admin panel 99.4% target 99% ✓
budget spent: 79% (34m 15s of 43m 12s)A single query in the repository is the outcome, and the fact that anybody can run it and get the same answer is the entire point of the exercise. The three conflicting numbers were not a measurement problem — they were four undecided questions.
What this costs
A number that constrains the roadmap, which is the point and is also the thing that makes it uncomfortable. The month the budget was exhausted on day nineteen, a feature slipped by a fortnight — and the policy held because it had been agreed in writing four months earlier by the person whose feature it was.
The monthly review is the piece most likely to lapse. It has no urgency, it competes with everything, and skipping it twice means the objectives describe a system that has changed. Putting it in the same hour as an existing recurring meeting is the only version that survived, and it is a compromise — a slot that shares an agenda gets the leftover twenty minutes.
It is also worth being honest that none of this made the system more reliable. It made the reliability measurable and made the trade explicit, which is a smaller claim than the term “SLO” usually carries and is the one that turned out to matter — the argument about whether to spend a quarter on resilience work stopped being a matter of opinion.