Every incident for three years had been resolved by the same person, which is not a rota — it is a single point of failure with a phone. The goal was not a functioning on-call rotation; it was that one other person could handle the eight things that actually happen.
The symptom
$ ./bin/incident-log --since=2021-05 --field=resolver | sort | uniq -c
44 alice
3 bob # all three during alice's holiday,
# all three escalated to alice
$ ./bin/incident-log --since=2021-05 --field=duration --resolver=alice
| ./bin/median
22m
$ ./bin/incident-log --since=2021-05 --field=duration --resolver=bob
| ./bin/median
3h 10mTwenty-two minutes against three hours and ten is not a difference in ability — it is a difference in access to context, and the three hours were mostly spent finding out where things are. Knowledge accrues to whoever responds, and responding is always faster than teaching.
Why it happens
The person who can fix it fastest gets called, which makes them faster, which makes them the person who gets called. Nothing about that loop is anybody’s decision and it tightens every time.
The fix
The inventory, from the log rather than from memory
47 incidents, three years, categorised:
12 a deploy broke something
9 the database was slow
7 a third party was down
5 a queue backed up
4 disk filled
2 a certificate expired
1 DNS
1 a memory leak in a worker
───
41 covered by eight failure modes
6 genuinely one-off
asked from memory, the same person listed five modes
and two of them had never happened.Eighty-seven per cent covered by eight documents is a much smaller number than it feels like from inside three years of responding. The discrepancy between the log and the recollection is the argument for using the log — memory over-weights the interesting incidents and under-weights the disk.
Runbooks written by the wrong person, on purpose
## The queue is backed up
**Confirm it is real**
1. `./bin/queue-depth` — above 1,000 for more than five
minutes is a backlog. A spike that clears is not.
2. `./bin/queue-depth --by-queue` — which one?
**The common cause: workers are not running**
3. `systemctl status 'worker@*'` — expect 6 active.
4. Fewer than 6: `systemctl start worker@{1..6}`.
Wait two minutes, re-check depth. If it is falling,
stop here and file a ticket about why they died.
**Six workers and still growing: one job is stuck**
5. `./bin/queue-inflight --older-than=5m`
6. ...the expert's version of step 1 was "check the queue".
the version written by somebody who did not know said
which command, which number, and for how long.
every runbook we had was written by the person who did
not need it, which is why none of them were usable.
inverting that is the whole trick and it is
uncomfortable for both people.The access problem
found by asking one person not to use their own laptop
for a day:
a registry token in a shell profile, created 2021,
not in the password manager
a database password in a saved connection
an ssh key with no passphrase and no second copy
a DNS provider login on a personal account
the last one is the hard one: a personal account at a
provider with no organisation feature means a recovery
process rather than a handover.
asked directly, the same person listed one of the four.The constraint — do not use your own machine — is what produces the real list, because none of these are secret and all of them are invisible to the person who has them. Asking what credentials exist produces a shorter and more optimistic answer every time.
The shadowing month, with the expert silent
the rules, agreed in advance:
the second person drives, always
the expert may answer a direct question and may not
volunteer anything
what the expert wanted to say goes in a document
afterwards, not during
a genuine outage suspends the rules
four weeks. the document accumulated 41 entries:
22 became runbook steps
6 became automation (a script, a check, an alert)
8 were context rather than procedure — why a thing
is the way it is. a separate document.
5 were preferences rather than knowledge.The silence is the mechanism. A helpful expert produces a shadowing month where nothing is learned and everything is watched, and writing down the interventions that were withheld is the artefact the exercise was actually for.
The six that became automation
a disk usage check with a 90% page and a 70% report
an alert on worker uptime under five minutes, which
catches a crash loop that systemd hides
a certificate expiry check on the SERVED certificate
rather than on the renewal job
a script that prints the last deploy, its commit and
who ran it
a one-line command that posts to the status page
a health endpoint that reports the deployed commit
all six existed as knowledge in one person's head and
as a step somebody would have had to think of.Verifying it worked
# two incidents, June and July, second person unaided
2024-06-14 queue backlog resolved 18m
(runbook step 4 — three workers had died
on a deploy)
2024-07-02 a third party down resolved 41m
(runbook: confirm, enable the degraded
path, notify, wait)
$ ./bin/incident-log --since=2024-06 --field=resolver
| sort | uniq -c
2 bob
1 alice
$ ./bin/credential-audit
credentials outside the password manager: 0
personal accounts with production access: 0Eighteen minutes against a three-hour median is the measurement, and the runbook step that resolved it is the one written by the person who then used it. The second incident took forty-one minutes and correctly did nothing except communicate — which is the harder thing to get right and is what the runbook was for.
What this costs
A rota of two is not a rota. It removes the case where one person is unavailable and does nothing about the case where both are, and it doubles the number of people whose evenings are interruptible. Calling it a fix would be dishonest — it is a bus factor of two, which is what a team of four can actually support.
The runbooks also decay. Eight documents describing systems that change weekly will be wrong within a year, and the only thing that keeps them honest is being used — which means the second person has to stay on the rota even when the first is available, and that is the discipline that will lapse first.