Last year’s handover produced a rota of two, which was honest about being insufficient. A third person joined in November and onboarding them was the test of whether the eight runbooks written in May were documentation or a description of what one person already knew.
The symptom
a rota of two, over eight months:
weeks where both were available 27
weeks where one was on leave 5
weeks where both were on leave 0 — by
arrangement,
which is the
problem
neither person had taken a fortnight off since the
previous May, and one of them had cancelled a holiday.A rota of two functions by two people coordinating their absences, which is not a rota — it is a constraint on two people’s lives. Nobody had raised it because nothing had failed.
Why it happens
Adding a person to a rota costs the existing members nothing in the moment and costs the new person a month, so the cost lands entirely on somebody who is not yet part of the arrangement. It is deferred until an absence forces it.
The fix
What the runbooks covered, and what they did not
eight runbooks, written in May by the person who did
not know the answers.
a deploy broke something used twice, worked
the database is slow used once, worked
a third party is down used once, worked
a queue backed up used twice, one
step wrong
disk filled not used
a certificate expired not used
DNS not used
a memory leak in a worker not used
four of eight have never been executed, which means
four of eight are hypotheses.A runbook that has been used is a procedure and one that has not is a guess, and the ratio here is even. The four unused ones are not necessarily wrong — they are unverified, which is a different claim and the one worth making out loud.
A drill rather than a document
announced a week in advance, run on a Tuesday at 14:00:
the scenario the supplier API returns 503 for 20
minutes, produced by an egress
firewall rule
the responder the new person, alone
the others watching, silent, taking notes
14:02 the rule goes in
14:06 the alert fires. good.
14:14 identified. the runbook's first step named a
dashboard that had been replaced in April.
14:16 the degraded path enabled
14:16 the status page: not updated. again.
14 minutes to mitigation, one runbook fix, one habit
that has now failed in three consecutive drills.Announcing in advance removes the surprise and removes the risk of a drill becoming a real incident, which is the right trade when the goal is exercising a procedure. The status page step has been the first step in the runbook since May and is still skipped, which suggests the problem is not the ordering.
The four things it found, all of them access
1 the log store: read access is granted to a group
the new person was not in.
2 the status page: a shared login in a password
manager vault they could not see.
3 the DNS provider: still a personal account,
eighteen months after this was first raised.
4 ssh: the certificate authority signs for
principals listed per host, and the new principal
was on two of four hosts.
none of these are knowledge. all four block a response,
and three of them were invisible to everybody who
already had the access.Access is what a knowledge-focused onboarding misses entirely, because the person teaching has all of it and cannot perceive its absence. The DNS account has now survived two attempts to fix it, both of which stalled on a billing arrangement that requires somebody outside engineering.
The handover note
## Handover, week of 2025-01-13
**Open**
- supplier B's webhook has been retrying since Friday.
ticket SUP-4182. no action unless still failing
Wednesday.
**Changed this week**
- the queue alert threshold is 60s, was 30s.
deliberate — see ADR-024.
**Watch**
- monthly invoicing runs Thursday 02:00. it has never
failed. if it does, runbook 6.Three sections and five minutes is what turns a rotation from handing over a pager into handing over context. The “watch” section is the one that earns its place — knowing something unusual is scheduled is the difference between a considered response and a cold start at two in the morning.
What a rota of three actually gives you
a fortnight off possible, for the first time
a week's rotation each person is primary one
week in three
an escalation path a secondary who is not the
same person every time
and what it does not give:
overnight coverage in any meaningful sense. three
people in one timezone is not a follow-the-sun
rotation and does not pretend to be.
the published response target remains "business hours,
best effort overnight", which is what three people can
honestly offer.Stating what the rota does not cover is the part that gets skipped, and it is what stops a customer conversation being had on the basis of a capability nobody has. Best effort overnight is a weaker promise than the previous arrangement implied and is the one that is actually true.
Verifying it worked
# three incidents, January to March, one per person
2025-01-22 queue backlog carol 16m
2025-02-14 a deploy rollback bob 9m
2025-03-03 supplier outage alice 38m
$ ./bin/incident-log --since=2025-01 --field=resolver
| sort | uniq -c
1 alice
1 bob
1 carol
$ ./bin/credential-audit
credentials outside the password manager: 0
personal accounts with production access: 1 ← DNS
# and the actual test
# one person took a fortnight in February. nothing
# was escalated to them.A fortnight with no escalation is the outcome the whole exercise was for, and it is a single data point. The DNS account remaining is the honest failure — it has been the finding of three consecutive audits and is the one thing engineering cannot resolve on its own.
What this costs
Three people’s evenings instead of two, and a rota that is still too small by any published standard. Three people in one timezone cannot offer overnight coverage and the honest response is to say so rather than to describe the arrangement in terms that imply otherwise.
The runbooks also decay, and four of eight remain unverified. A drill exercises one scenario a quarter, which at that rate takes two years to cover the set — and the ones that never come up in practice are exactly the ones a drill has to reach, because nothing else will.