A restore drill that had been run monthly by the person who wrote it, handed to somebody who had not.
the drill script worked. the process did not:
the key was in a password manager vault the second
person had no access to
the runbook said "decrypt the backup" and not how
the scratch database's port conflicted with their
local one, which the script did not surface
the assertions passed and they had no way to know
what they meant
time: 3 hours, against a documented 30 minutes.
A drill run by its author verifies the mechanism and nothing about the procedure. Three hours against a documented thirty minutes is the honest recovery time for the realistic case, and the recovery objective had been quoted from the wrong number for a year.