A deploy at 16:40 broke checkout for about eight per cent of customers in a way the smoke tests did not cover. The decision to roll back was made at 16:47 and the previous version was live at 17:06, because rolling back meant checking out the old tag and running the build again.
The symptom
$ time ./deploy.sh --ref=v2021.12.3
composer install --no-dev 4m12s
npm ci && npm run build 6m48s
docker build 5m02s
push, pull, restart 2m41s
real 18m43s
# and worse: the rebuild is not guaranteed to produce what
# was running — the base image tag under the lock files
# may not resolve to the same image.Nineteen minutes is the number and it is not the worst property. The rebuilt artefact is only probably identical to what was running an hour earlier, because a floating base image tag, a package mirror and a build-time timestamp all vary independently of the lock files.
Why it happens
A deploy pipeline that builds and releases in one motion has no artefact to go back to — the output of the build was pushed and the previous one may still exist in a registry or may have been pruned. Rolling back therefore means rebuilding, which is the slowest thing in the pipeline.
The fix
Separating build from release
before: build → push → restart, as one command
after: BUILD an immutable artefact, pushed, and its
digest recorded
RELEASE points production at a digest — no build,
no install, no compilation
rolling back is then the same operation with an older
digest, which is the whole point.jobs:
build:
outputs: { digest: "${{ steps.push.outputs.digest }}" }
steps:
- uses: docker/build-push-action@v2
id: push
with: { push: true, tags: "registry/app:${{ github.sha }}" }
release:
needs: build
environment: production
steps:
- run: ./bin/release "${{ needs.build.outputs.digest }}"Releasing by digest rather than by tag is what makes the artefact unambiguous — a tag can be moved and a digest cannot, so the release log records exactly which bytes were running. It also means a rollback is a digest from the release log rather than a guess about which tag was current.
The release, and the state that is not swapped
docker pull "registry/app@$digest"
./bin/precheck-schema "$digest" # refuse if incompatible
docker compose up -d --no-deps app
./bin/smoke --wait=60 || ./bin/release "$(./bin/previous-digest)"
# and the shared state a swap does NOT swap: the database
# schema (the hard one), uploaded files, session data, the
# cache namespace, and jobs serialised by the OLD codeThe automatic rollback on a failing smoke test is what makes the ninety seconds real, and it is only safe because the release is a digest swap with no build. A rollback that takes nineteen minutes cannot be automatic, because nobody will let a script start one unattended.
The schema precheck refusing to release an image whose migrations do not match the current schema is what stops a rollback making things worse — going back to code that expects a column that has been dropped is a second outage on top of the first.
Migrations, which are why rollback is hard
a deploy is reversible. a migration is not.
ADD COLUMN nullable reversible, safe to leave
ADD INDEX reversible, safe to leave
DROP COLUMN IRREVERSIBLE — the data is gone
RENAME COLUMN breaks the old code instantly
the rule that makes rollback possible: a deploy may contain
only migrations the PREVIOUS version tolerates. everything
else is expand-contract, across two releases.Writing that rule down and enforcing it in review is the single change that makes rollback a real option rather than a plan. Without it, every deploy containing a destructive migration is a deploy that cannot be undone, and nobody notices until the one that needs undoing.
// enforced, so it is not a matter of remembering
public function testPendingMigrationsAreBackwardCompatible(): void
{
$destructive = ['dropColumn', 'renameColumn', 'drop('];
foreach ($this->pendingMigrationFiles() as $file) {
foreach ($destructive as $needle) {
$this->assertStringNotContainsString($needle,
file_get_contents($file),
"{$file}: use expand-contract, or @allow-destructive");
}
}
}A string match on migration source is crude and catches the cases that matter, and the escape hatch in the message is deliberate — a destructive migration is sometimes correct, and it should require a comment that a reviewer sees rather than being impossible.
The other shared state
// a cache namespace that includes the release, so a
// rollback does not read the new version's cached shapes
'prefix' => 'app:' . config('app.release') . ':',
// and the queue: a job serialised by v4 and processed by
// v3 fails to deserialise. jobs carry a schema version,
// and an unknown one is RELEASED BACK with a delay
// rather than failed.The queue is the piece that is forgotten in every rollback plan and produces a failure ten minutes after the rollback appears to have worked. Releasing an unrecognised job back onto the queue rather than failing it means the jobs enqueued by the new version survive until it is deployed again.
Namespacing the cache by release means a rollback starts cold, which is a real cost — a cold cache after a rollback is a load spike at exactly the wrong moment. The alternative is a rollback that reads objects cached in a shape the old code cannot parse, which is worse and harder to diagnose.
Rehearsing it, on a schedule
# monthly, in working hours, on production
$ ./bin/rollback-drill
schema precheck compatible
release 14s
smoke 38s
rolled forward 21s
total: 1m13s
# what the first three drills found:
# 1 previous-digest returned the wrong one after a
# failed release
# 2 the cache prefix was not release-scoped
# 3 the smoke test did not cover checkoutThree of the first three drills found something broken, which is the expected outcome and the reason for doing it in working hours with everybody watching. A rollback mechanism that has never been exercised is a rollback mechanism that does not work, and finding that out during an incident is how nineteen minutes becomes an hour.
Verifying it worked
$ ./bin/release "$(./bin/previous-digest)"
pulled sha256:8c9e114… in 9s; precheck compatible;
released in 14s; smoke 38s ok
real 1m11s # was 18m43s
$ ./bin/release-log | head -3
2021-12-21 16:47 sha256:8c9e114… rollback
2021-12-21 16:40 sha256:a41f2b8… deploy
2021-12-20 11:02 sha256:3e11fa4… deployThe release log recording who released which digest and when is what makes an incident timeline reconstructable, and it costs one append per release. It also answers “what was running at 16:45” without anybody guessing.
What this costs
Two-phase schema changes, forever. Every column rename becomes two releases with a backfill between them, every drop waits a week, and the migration safety test will occasionally block something that is genuinely fine. That is a permanent tax on schema work in exchange for a rollback that takes ninety seconds instead of nineteen minutes.
The cold cache after a rollback is the other cost and it is a load spike at the worst moment — a rollback happens because something is already wrong, and it arrives with an empty cache. Warming the critical keys as part of the release script mitigates it and adds twenty seconds, which was worth it and is a number that will grow.
And the honest limit: none of this helps with a change that has already written bad data. A rollback restores the code and not the rows, so a deploy that corrupted something is a restore rather than a rollback, and that is still measured in hours. The ninety seconds covers the common case, which is a deploy that is broken rather than destructive, and it is worth being clear about which one is being planned for.