Türker Yıldırım

A deploy that can be undone in ninety seconds

Deep dive10 min read

A deploy at 16:40 broke checkout for about eight per cent of customers in a way the smoke tests did not cover. The decision to roll back was made at 16:47 and the previous version was live at 17:06, because rolling back meant checking out the old tag and running the build again.

The symptom

$ time ./deploy.sh --ref=v2021.12.3
  composer install --no-dev        4m12s
  npm ci && npm run build          6m48s
  docker build                     5m02s
  push, pull, restart              2m41s
real	18m43s

# and worse: the rebuild is not guaranteed to produce what
# was running — the base image tag under the lock files
# may not resolve to the same image.

Nineteen minutes is the number and it is not the worst property. The rebuilt artefact is only probably identical to what was running an hour earlier, because a floating base image tag, a package mirror and a build-time timestamp all vary independently of the lock files.

Why it happens

A deploy pipeline that builds and releases in one motion has no artefact to go back to — the output of the build was pushed and the previous one may still exist in a registry or may have been pruned. Rolling back therefore means rebuilding, which is the slowest thing in the pipeline.

The fix

Separating build from release

before:  build → push → restart, as one command

after:   BUILD    an immutable artefact, pushed, and its
                  digest recorded
         RELEASE  points production at a digest — no build,
                  no install, no compilation

rolling back is then the same operation with an older
digest, which is the whole point.
jobs:
  build:
    outputs: { digest: "${{ steps.push.outputs.digest }}" }
    steps:
      - uses: docker/build-push-action@v2
        id: push
        with: { push: true, tags: "registry/app:${{ github.sha }}" }

  release:
    needs: build
    environment: production
    steps:
      - run: ./bin/release "${{ needs.build.outputs.digest }}"

Releasing by digest rather than by tag is what makes the artefact unambiguous — a tag can be moved and a digest cannot, so the release log records exactly which bytes were running. It also means a rollback is a digest from the release log rather than a guess about which tag was current.

The release, and the state that is not swapped

docker pull "registry/app@$digest"
./bin/precheck-schema "$digest"    # refuse if incompatible
docker compose up -d --no-deps app
./bin/smoke --wait=60 || ./bin/release "$(./bin/previous-digest)"

# and the shared state a swap does NOT swap: the database
# schema (the hard one), uploaded files, session data, the
# cache namespace, and jobs serialised by the OLD code

The automatic rollback on a failing smoke test is what makes the ninety seconds real, and it is only safe because the release is a digest swap with no build. A rollback that takes nineteen minutes cannot be automatic, because nobody will let a script start one unattended.

The schema precheck refusing to release an image whose migrations do not match the current schema is what stops a rollback making things worse — going back to code that expects a column that has been dropped is a second outage on top of the first.

Migrations, which are why rollback is hard

a deploy is reversible. a migration is not.

  ADD COLUMN nullable   reversible, safe to leave
  ADD INDEX             reversible, safe to leave
  DROP COLUMN           IRREVERSIBLE — the data is gone
  RENAME COLUMN         breaks the old code instantly

the rule that makes rollback possible: a deploy may contain
only migrations the PREVIOUS version tolerates. everything
else is expand-contract, across two releases.

Writing that rule down and enforcing it in review is the single change that makes rollback a real option rather than a plan. Without it, every deploy containing a destructive migration is a deploy that cannot be undone, and nobody notices until the one that needs undoing.

// enforced, so it is not a matter of remembering
public function testPendingMigrationsAreBackwardCompatible(): void
{
    $destructive = ['dropColumn', 'renameColumn', 'drop('];

    foreach ($this->pendingMigrationFiles() as $file) {
        foreach ($destructive as $needle) {
            $this->assertStringNotContainsString($needle,
                file_get_contents($file),
                "{$file}: use expand-contract, or @allow-destructive");
        }
    }
}

A string match on migration source is crude and catches the cases that matter, and the escape hatch in the message is deliberate — a destructive migration is sometimes correct, and it should require a comment that a reviewer sees rather than being impossible.

The other shared state

// a cache namespace that includes the release, so a
// rollback does not read the new version's cached shapes
'prefix' => 'app:' . config('app.release') . ':',

// and the queue: a job serialised by v4 and processed by
// v3 fails to deserialise. jobs carry a schema version,
// and an unknown one is RELEASED BACK with a delay
// rather than failed.

The queue is the piece that is forgotten in every rollback plan and produces a failure ten minutes after the rollback appears to have worked. Releasing an unrecognised job back onto the queue rather than failing it means the jobs enqueued by the new version survive until it is deployed again.

Namespacing the cache by release means a rollback starts cold, which is a real cost — a cold cache after a rollback is a load spike at exactly the wrong moment. The alternative is a rollback that reads objects cached in a shape the old code cannot parse, which is worse and harder to diagnose.

Rehearsing it, on a schedule

# monthly, in working hours, on production
$ ./bin/rollback-drill
  schema precheck  compatible
  release          14s
  smoke            38s
  rolled forward   21s
  total: 1m13s

# what the first three drills found:
#   1  previous-digest returned the wrong one after a
#      failed release
#   2  the cache prefix was not release-scoped
#   3  the smoke test did not cover checkout

Three of the first three drills found something broken, which is the expected outcome and the reason for doing it in working hours with everybody watching. A rollback mechanism that has never been exercised is a rollback mechanism that does not work, and finding that out during an incident is how nineteen minutes becomes an hour.

Verifying it worked

$ ./bin/release "$(./bin/previous-digest)"
pulled sha256:8c9e114… in 9s; precheck compatible;
released in 14s; smoke 38s ok

real	1m11s        # was 18m43s

$ ./bin/release-log | head -3
2021-12-21 16:47  sha256:8c9e114…  rollback
2021-12-21 16:40  sha256:a41f2b8…  deploy
2021-12-20 11:02  sha256:3e11fa4…  deploy

The release log recording who released which digest and when is what makes an incident timeline reconstructable, and it costs one append per release. It also answers “what was running at 16:45” without anybody guessing.

What this costs

Two-phase schema changes, forever. Every column rename becomes two releases with a backfill between them, every drop waits a week, and the migration safety test will occasionally block something that is genuinely fine. That is a permanent tax on schema work in exchange for a rollback that takes ninety seconds instead of nineteen minutes.

The cold cache after a rollback is the other cost and it is a load spike at the worst moment — a rollback happens because something is already wrong, and it arrives with an empty cache. Warming the critical keys as part of the release script mitigates it and adds twenty seconds, which was worth it and is a number that will grow.

And the honest limit: none of this helps with a change that has already written bad data. A rollback restores the code and not the rows, so a deploy that corrupted something is a restore rather than a rollback, and that is still measured in hours. The ninety seconds covers the common case, which is a deploy that is broken rather than destructive, and it is worth being clear about which one is being planned for.