Eight failure modes covering forty-one incidents

Three years of incident notes, read as an inventory rather than as history.

47 incidents, categorised:

  12  a deploy broke something         runbook
   9  the database was slow           runbook
   7  a third party was down          runbook
   5  a queue backed up               runbook
   4  disk filled                     runbook
   2  a certificate expired           runbook
   1  DNS                             runbook
   1  a memory leak in a worker       runbook
  ───
  41  covered by eight runbooks
   6  genuinely one-off

Eight documents cover eighty-seven per cent of what has ever gone wrong, which is a much smaller number than it feels like from inside the experience of responding to incidents. The six one-offs are the argument against trying to cover everything — a ninth runbook would have covered one incident from 2022 that cannot recur.