The deploy was a twenty-two step runbook that one person had ever executed successfully. Four of the steps said some version of “check it looks right”, which is not a step, and the person who could run it had booked three weeks off in August.
The symptom
the runbook, abbreviated:
1 merge to main
2 wait for CI
3 ssh to app-1
4 cd /srv/app && git pull
5 composer install --no-dev
6 npm ci && npm run build
7 php artisan migrate ← sometimes
8 php artisan cache:clear ← if config changed
9 php artisan queue:restart ← always, and often
forgotten
...
18 check the site looks right
19 check the queue is moving
20 check the error rate in Grafana
21 if wrong, git checkout the previous tag and
repeat 5-9
22 tell #generalSteps seven and eight are conditional on a judgement, step nine is unconditional and gets forgotten, and step twenty-one is a rollback described as “repeat some earlier steps” — which under pressure is where somebody runs the migration again.
Why it happens
A deploy grows by accretion. Each step was added the day something went wrong, by the person who fixed it, and nobody is ever asked to re-derive the whole thing from scratch. The result is correct and can only be executed by the person who has the context in their head.
The fix
Writing down what actually happens
the undocumented steps, found by watching:
before step 5 a `git status` to check nothing was
edited on the server. it usually had
been.
during step 6 npm run build occasionally fails on
memory and is re-run. nobody knew.
after step 7 a manual check that the migration
did not lock. the check is "watch
the site".
after step 9 a 10-second pause, because restarting
the queue too soon after the code
changes drops a job.
four steps that existed only in one person's hands.Watching somebody deploy without speaking is the technique, and it produced more information than an interview would have. The ten-second pause was the most interesting: it is a workaround for a real race, held entirely as muscle memory.
The three load-bearing manual steps
cache clear config is cached in production, so
a changed .env or config file is
invisible until the cache is rebuilt.
conditional on "did config change",
which nobody could determine reliably.
→ always rebuild. it costs 400ms.
migration conditional on "are there new
migrations", which the tool can
answer. → always run; it is a no-op
when there is nothing to do.
queue restart unconditional and forgotten. → part
of the script, after the symlink
switch, with the pause.Two of the three conditions were a human doing a job a command could do, and the third was not a condition at all. Removing a judgement from a procedure is nearly always worth a small unconditional cost.
Migrations that are safe before the code
the ordering problem: the migration runs while the old
code is still serving.
safe before add a nullable column
add a table
add an index (online)
widen a column
NOT safe before drop a column
rename anything
add a NOT NULL column with no default
narrow a type
the rule adopted: a migration must be safe against BOTH
the old and the new code. anything that is not gets
split across two releases — expand, then contract.// release 1: expand
Schema::table('orders', fn (Blueprint $t) => $t->string('region', 2)->nullable());
// code writes both old and new, reads old
// release 2: backfill (a job, not a migration)
// release 3: contract
// code reads new only; the old column is droppedExpand-and-contract is three releases where the naive version is one, and it is the only pattern that makes a zero-downtime deploy possible with a schema change. Writing the rule down is what stops the fourth person discovering it during an incident.
The queue restart, and the job mid-flight
# the symlink switch is atomic; the workers are not
ln -sfn "$release" /srv/app/current.new
mv -Tf /srv/app/current.new /srv/app/current
# tell the workers to finish the current job and exit
sudo -u www-data /srv/app/current/artisan queue:restart
# the workers poll for this flag between jobs, so a job
# in flight completes against the OLD code — which is
# why expand-and-contract matters for the queue too.
systemctl reload php-fpmA worker finishing its current job against the old code is correct behaviour and is the reason a job payload must be readable by both versions. The ten-second pause in the original runbook was compensating for a supervisor configuration that killed workers rather than signalling them, which was fixed rather than preserved.
One command, with the checks inline
#!/usr/bin/env bash
set -euo pipefail
ref="${1:?usage: deploy <git-ref>}"
previous=$(readlink -f /srv/app/current)
log() { printf ' 33[1m==> %s 33[0mn' "$*"; }
fail() { printf 'FAILED: %sn' "$*" >&2; rollback; exit 1; }
rollback() {
log "rolling back to $previous"
mv -Tf <(echo) /dev/null 2>/dev/null || true
ln -sfn "$previous" /srv/app/current.new
mv -Tf /srv/app/current.new /srv/app/current
systemctl reload php-fpm
}
log "building $ref"
build_release "$ref" || fail 'build'
log 'migrating'
run_migrations || fail 'migrations'
log 'switching'
switch_release && systemctl reload php-fpm
log 'health'
wait_healthy 20 || fail 'health check'
log 'workers'
restart_workers
log "deployed $ref"The rollback is a function called from the failure path rather than a paragraph in a document, which is the structural change. Every fail restores the previous release before exiting, so the worst outcome of a broken deploy is the previous version and a non-zero exit code.
The health check that is not a ping
Route::get('/health/deep', function (Connection $db, Repository $cache) {
return response()->json([
'commit' => config('app.commit'),
'db' => $db->select('SELECT 1') ? 'ok' : 'fail',
'cache' => $cache->get('health-probe') !== null ? 'ok' : 'fail',
'migrations' => app(Migrator::class)->pendingCount() === 0 ? 'ok' : 'pending',
]);
});Returning the deployed commit is what makes the check assert the right thing: the script compares it against the ref it just deployed, so a health check served by a stale process fails rather than passing. That is the failure the original runbook’s “check it looks right” was supposed to catch.
Handing it over, and watching without speaking
somebody who had never deployed this application:
attempt 1 failed at the ssh config step. the
runbook assumed a host alias that only
existed on one laptop.
attempt 2 failed on a sudo permission. the deploy
user needed one systemctl rule that had
been granted by hand in 2021.
attempt 3 succeeded. 4 minutes 10 seconds.
attempt 4 deliberate rollback. 40 seconds.
two failures, both configuration that lived on one
machine, both fixed in the provisioning scripts.Neither failure was in the deploy script. Both were environment assumptions that were invisible to the person who had them, which is exactly what handing it to a stranger is for — the script was fine and the surrounding assumptions were not.
Verifying it worked
# four deploys, three people, one week
alice 4m 10s ok
bob 3m 58s ok
bob 4m 02s rolled back deliberately, 41s
carol 4m 21s ok
# and the August test: three weeks, eleven deploys,
# none by the person who wrote the original runbook
$ git log --oneline --grep='deploy:' --since=2023-08-01 | wc -l
11
$ grep -c FAILED /var/log/deploy.log
1 # a genuine failing test, caught by the pipelineEleven deploys in three weeks with the original operator away is the outcome the whole exercise was for. The one failure was a test failing before anything was switched, which is the pipeline working rather than the deploy failing.
What this costs
A script that is now the only thing that knows how the system starts. The runbook was bad and it was readable; the script is good and requires somebody to read bash to understand what a deploy does. That is a fair trade and it is a trade.
The expand-and-contract rule is the ongoing cost and it is the one that will be violated. Three releases to rename a column is genuinely tedious, and the pressure to do it in one arrives on the day somebody is in a hurry — which is the day it will not work.
There is also a category of failure the script cannot cover: something that is only wrong under real traffic, minutes after the switch. The health check runs for twenty seconds against one process, and a canary or a gradual rollout is the answer to that — which is a considerably larger piece of work and was explicitly deferred.