An ssh certificate authority, one year on

A year of eight-hour certificates rather than an authorized_keys file, and the operational reality of it.

what worked:
  onboarding a third person: one signing command
  offboarding: nothing to do. the certificate expired.
  the audit trail: every signature logged with a
    principal and a validity window

what did not:
  a certificate expiring mid-incident, twice. the
    renewal is a command and nobody wants to run one
    at 03:00.
  → a 24-hour validity for the on-call principal,
    which is a deliberate weakening and is written down.

and the thing nobody thought about: the CA key's own
backup, which lived in the same place as everything else.

The mid-incident expiry is the failure mode that theory does not predict, and the answer is a longer window for a specific principal rather than a longer window for everybody. The CA key being backed up alongside everything it protects is the more serious finding and was fixed by the offsite work in December.