For two years one person had the phone, because he had built most of it and could fix anything at two in the morning. He gave notice in October, and the handover conversation established that nobody else had ever been woken up by this system or knew what to do if they were.
The symptom
the runbook, in full:
"If the site is down, restart php-fpm. If that does not
work, call the person whose phone it is."
what was actually known, and only by one person:
which alerts matter and which are noise
that the 03:00 disk alert is the backup job, always
that the queue depth alert means the payment gateway,
not the queue
the sudo password on the database host
that the failover has never been testedThe bus factor is the obvious problem and it is not the interesting one. The interesting one is that two years of operational knowledge existed only as one person’s judgement, and none of it had ever had to be explained.
Why it happens
A rotation of one is the path of least resistance at every individual moment. The person who can fix it fastest gets called, fixing it is quicker than documenting it, and each of those decisions is correct in isolation.
A team of four makes the arithmetic hard in a way a team of ten does not. One week in four is frequent enough to be a real imposition and infrequent enough that nobody develops fluency — three weeks off is long enough to forget where the dashboard is.
The fix
Deciding what actually pages, first
before: 31 alerts, all to the same channel, all at any hour.
11 of them fired weekly. none had ever been actioned.
the filter applied to every one:
is a human needed? no → it is a dashboard, not
an alert
is a human needed NOW? no → it is a ticket
is it actionable at 3am? no → it waits until 9
has it ever been real? no → delete it
after: 6 page. 9 notify during hours. 16 deleted.This is the work that has to happen before a rotation, not after, because putting three more people on a rotation that pages eleven times a week for nothing is how a rotation gets abandoned in a month. Every alert that fires and is ignored trains everybody to ignore alerts.
# and the six that remained are symptom-based, not cause-based
- alert: CheckoutFailing
expr: |
sum(rate(http_requests_total{route="checkout",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{route="checkout"}[5m])) > 0.05
for: 3m
labels: { severity: page }
annotations:
summary: '{{ $value | humanizePercentage }} of checkouts failing'
runbook: 'https://wiki/runbooks/checkout-failing'
# not: 'php-fpm process count is low', which is a cause and
# might be fine.Alerting on the symptom rather than the cause is what takes thirty-one alerts to six, because most causes have no consequence — a process count that dips and recovers pages nobody if the alert is about whether checkout works.
The runbook link in the annotation is the difference between an alert and a useful alert. A person woken at three in the morning should not have to search a wiki, and an alert without a runbook is a decision deferred to somebody who is asleep.
Runbooks written by the person who did not build it
# Checkout failing
## What this means
More than 5% of checkout requests returned 5xx for 3 minutes.
Customers are seeing errors at payment.
## First, check
1. Grafana → Checkout → is it the gateway or us?
https://grafana/d/checkout
2. `kubectl logs -l app=checkout --since=10m | grep -i error`
## If it is the gateway
The status page: https://status.provider.example
We queue and retry automatically for 30 minutes. If it will
be longer, enable the banner:
`wp option update turkerdev_payment_banner 1`
## If it is us
The most common cause is the connection pool exhausted.
`SHOW PROCESSLIST` on db-01, look for sleeping connections.
Restarting php-fpm clears them: `systemctl restart php-fpm`
## If none of the above
Escalate. Rota is in the pinned message. It is fine to escalate.Having the runbook written by somebody who did not build the system is what makes it useful, because the person who built it writes down the parts they find interesting and omits the parts they find obvious — which are exactly the parts that are needed at three in the morning.
The last line matters more than it looks. A rotation where escalating feels like failure is a rotation where somebody spends two hours on something the expert would fix in ten minutes, and saying so explicitly in every runbook is the cheapest way to prevent it.
The rotation itself, for four people
tried, and did not work:
one week each, straight three weeks off is enough to forget
everything. every shift re-learned.
primary + secondary four people cannot sustain two
rotations. it is one week in two.
what worked: one week each, and the person coming OFF is the
escalation for the person coming ON. each shift has continuity;
each person is engaged two weeks in four, one passively. plus
a monthly game day, so the habit does not decay.The outgoing person as escalation is the arrangement that solved the fluency problem without doubling the load. They have the context from the week just finished, they are not being paged directly, and the handover happens because there is a reason for it rather than because a process says so.
The game day is what keeps the runbooks true. Breaking something on staging deliberately, in working hours, with the on-call person following the runbook, finds the step that no longer works — and it found four of them in the first session.
The parts that are not technical
written down before the first shift, or they are decided at
3am by somebody guessing:
compensation a flat weekly rate, plus hourly after midnight.
agreed with the business, in writing.
time back paged after midnight → the next morning is off.
not "if you need it". automatic.
saying no a shift can be handed over, no reason required.
the rota is a spreadsheet, not a contract.
scope the six alerts. a customer email at 9pm is not
an incident.The compensation conversation has to happen before the rotation starts and with the business rather than within the team, because a rotation introduced without it is a change to four people’s working conditions decided by their colleague. That is the fastest way to have the whole thing resented.
The automatic morning off is the one that gets quietly skipped, and skipping it is how a rotation becomes a reason to leave. Making it automatic rather than available removes the judgement about whether it was bad enough to claim.
Verifying it worked
the first quarter:
pages, total 7 outside hours 2 false positives 1
escalations 3 runbook right 5 runbook wrong 2
before: ~11 alerts/week to one person, none actioned.
and the one that mattered: the person who had held the phone
for two years left in January, and the system was on-called by
three people who had never been woken by it before November.Three escalations out of seven is a healthy number rather than a bad one — it means people escalated rather than struggling, which is the behaviour the runbooks explicitly ask for. A quarter with zero escalations would suggest somebody is not asking for help.
The two wrong runbooks are the most valuable output of the quarter. Both were wrong in the same way — a command that had changed when the deployment moved — and neither would have been discovered by anyone reading them.
What this costs
Money, in the compensation, and a real imposition on three people who did not previously have one. That is the honest accounting: the previous arrangement was cheaper because one person absorbed the entire cost, and spreading it makes it visible rather than creating it.
A four-person rotation is also genuinely at the lower limit of what is sustainable, and the arrangement here works because the system pages seven times a quarter. The same rotation on a system that pages twice a week would burn through the team in six months, and the correct response to that is to fix the system rather than to add a fifth name — which is easy to write and hard to hold to when the alerts are firing.