The backup that had to leave the building

Backups have been running nightly to one provider’s object storage since 2020, encrypted, versioned, and restored monthly in a drill. The arrangement is sound and it assumes the disaster is local — the failure modes it does not cover are the ones where the problem is the provider or the credential.

The symptom

the failure modes, ranked by likelihood rather than
by drama:

  1  a compromised credential, buckets emptied
  2  a mistake — a lifecycle rule with the wrong
     prefix, a script with a typo
  3  a provider outage of hours
  4  a provider terminating the account
  5  a provider ceasing to exist

the current arrangement survives 3 and 5 badly, and 1
and 2 not at all — which are the likely ones.

The dramatic failure — a provider going out of business — is the one everybody plans for and the least likely. A credential with delete permission and a script with a wrong prefix are the realistic ways backups disappear, and versioning does not help when the versions are deleted too.

Why it happens

One provider is simple and a second one is a project, so the second one is deferred until there is a reason. The reason is always hypothetical until it is not.

The fix

Object lock, which addresses the likely cases

aws s3api put-object-lock-configuration 
  --bucket turkerdev-backups-offsite 
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 35 }
    }
  }'

COMPLIANCE mode: not even the account root can delete
an object before the retention expires.

which means:
  a compromised credential cannot destroy backups
  a lifecycle rule with a wrong prefix cannot either
  and neither can we, for 35 days, deliberately

the storage cost is bounded — 35 days of dailies —
and the irreversibility is the feature.

A write-only credential protects against a compromised backup process and not against a compromised account, which is the failure mode that actually destroys backups. Compliance mode is the only thing that survives an attacker with full account access, and the cost is that a genuine mistake is also unfixable for thirty-five days.

The second provider

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject"],
    "Resource": "arn:aws:s3:::turkerdev-backups-offsite/*"
  }]
}

One action, one resource, and no list, get or delete — a credential that can write a backup and cannot read one, enumerate them or remove any. The restore uses a separate credential held by two people and not present on any host, which is the arrangement that makes the write-only key safe to have on a server.

The key that lives in neither provider

  the backup is encrypted with age, using a public key
  the private key is held:
    primary    the password manager, three people
    secondary  a sealed envelope, offsite, with the
               recovery procedure printed alongside

  and NOT in either storage provider, because a key
  stored beside the thing it protects is not a key.

the drill alternates: two of the four restores in the
last year used the envelope.

Alternating the drill between key sources is the part that is easy to skip and is the entire point of the secondary — a copy that has never been used is a copy that might be a photograph of the wrong screen. It has been correct both times, which is two data points.

The restore drill, from the second copy

  primary copy    download  4m  decrypt 1m  import 22m
  offsite copy    download 41m  decrypt 1m  import 22m

64 minutes against 27.

the difference is entirely the download: the second
provider is in a different region and the restore host
is not.

and the recovery objective had been quoted as "about
thirty minutes" — which was true for the copy that the
likely failure modes destroy.

A recovery objective measured against the fast path is a number for the case that will not happen, which is the general problem with measuring the convenient thing. Sixty-four minutes is the honest figure and is now the one in the document, with a note that a restore host in the second region would halve it and has not been built.

The egress bill

  storage, provider A, 35 days of dailies    £6.40
  storage, provider B, 35 days               £7.10
  egress A → B, full copies nightly         £41.00
  ─────────────────────────────────────────────────
                                            £54.50

after switching to incrementals with a weekly full:

  egress                                     £8.20
  total                                     £21.70

and the cost of that: restore time goes up with the
chain length. measured at 64 minutes with a chain of
six, against 51 with a full.

Egress pricing makes a naive second copy cost several times the storage, and incrementals with periodic fulls is the same answer that has always applied to backups. The restore time going up with the chain is the trade and it is measured rather than assumed, which is what the monthly drill is for.

Verifying it worked

$ aws s3api get-object-lock-configuration 
    --bucket turkerdev-backups-offsite
  { "ObjectLockEnabled": "Enabled", ... COMPLIANCE, 35 }

# a deliberate deletion attempt, with the root credential
$ aws s3api delete-object --bucket turkerdev-backups-offsite 
    --key db/2025-11-20.sql.age
  An error occurred (AccessDenied): Object is subject to
  a Compliance mode retention period

# a simulated key compromise: the write-only credential
$ aws s3 ls s3://turkerdev-backups-offsite/ --profile backup-writer
  An error occurred (AccessDenied)

$ ./bin/restore-drill --source=offsite --key=envelope
  64m 10s, all assertions passed

Attempting the deletion with the most privileged credential available is the assertion that compliance mode is doing what it claims, and it is the only way to know. Running the drill with the envelope key rather than the password manager is the other half — both of those are ten minutes and both are the difference between a plan and a hope.

What this costs

Two of everything, and a key that is now the single point of failure. Both copies are encrypted with the same key, so losing it loses both — which is a concentration of risk created by the very arrangement that spreads the storage risk, and the sealed envelope is the entire mitigation.

Compliance mode also means a genuine mistake is unfixable for thirty-five days. A backup containing something that should not have been backed up — a database copy with data a customer has asked to be deleted — cannot be removed, which is a real conflict with a deletion obligation and has no good answer except not backing it up in the first place.