The deploy has been a single command since 2023 and has been run by a person, roughly an hour after merge, by whoever noticed. Making it run on merge is a one-line change to a workflow file, and it took two years because the change is not the automation.
The symptom
eight weeks, measured:
merges 64
deploys 41
median merge-to-deploy 1h 40m
p90 6h 20m
merges batched into one
deploy 23
and what the batching costs: a rollback reverts four
changes, three of which were fine.Batching is the real cost rather than the latency. A deploy containing four merges is a deploy whose failure cannot be attributed, and the response to a problem is to revert all of it — which is why the p90 matters more than the median.
Why it happens
A deploy that requires a decision gets deferred until somebody is confident enough to make it, and confidence comes from the tests, the rollback and the monitoring rather than from the automation. Automating the trigger before those exist moves the risk rather than removing it.
The fix
The four gates, each of which predates this change
1 a test suite at 90 seconds (2024-09)
fast enough that it runs before every push, so a
merge has already been verified locally and in CI.
2 an automatic rollback on a health check (2023-05)
a deploy that produces a failing health check
reverts within 40 seconds, unattended.
3 the expand-and-contract migration rule (2023-05)
a migration must be safe against both the old and
the new code, checked by a script.
4 per-consumer error rate alerting (2024-06)
a change that breaks one integrator is visible
within five minutes rather than at the next
support ticket.
none of these were built for this. all four are the
reason it is now possible.Continuous deployment is a consequence of four unrelated pieces of work rather than a project of its own, which is why it took two years and one line. Anybody attempting the one line first would have automated the deployment of unverified changes with no rollback.
The migration safety check
// runs against the pending migrations, in CI
private const array UNSAFE = [
'/DROPs+COLUMN/i' => 'dropping a column',
'/RENAMEs+COLUMN/i' => 'renaming a column',
'/MODIFYs+COLUMN.*NOT NULL/i' => 'adding NOT NULL',
'/DROPs+TABLE/i' => 'dropping a table',
];
// anything matching requires an explicit marker in the
// migration saying the previous release stopped using itthe marker, which is the whole mechanism:
/**
* @contract-safe The `legacy_ref` column stopped
* being written in release 5.4.0 and stopped being
* read in 5.4.2. Both are deployed.
*/
without it, the check fails and the merge is blocked.
with it, a human has asserted something specific that
a reviewer can verify.A pattern match on SQL is crude and it is the only thing that scales to every migration without a human reading each one. The marker turns the check from an obstacle into a prompt — the author has to state which release stopped using the column, which is a claim somebody can check against the deployment history.
The deploy window, which is a business constraint
deploy:
needs: [test, analyse, migration-safety]
if: |
github.ref == 'refs/heads/main' &&
github.event_name == 'push'
steps:
- name: within the deploy window
run: |
h=$(TZ=Europe/London date +%-H)
d=$(TZ=Europe/London date +%u)
[ "$d" -le 4 ] && [ "$h" -ge 8 ] && [ "$h" -lt 16 ] || {
echo 'outside the deploy window; queued'
exit 78
}Monday to Thursday, eight until four, is a constraint about who is available rather than about the software — a deploy at five on a Friday is fine technically and is a deploy nobody will notice failing. Exit code 78 marks the job as neutral rather than failed, so a merge outside the window is not a red build.
What stayed manual
a migration marked @contract-safe by its author
→ deploys automatically. the marker is the review.
a migration the check flags with no marker
→ the merge is blocked. not the deploy.
anything touching the schema in a way the pattern
match cannot classify — a data backfill, a
partition operation, an ALTER on a table over 10M
rows
→ labelled `manual-deploy`, which skips the
automatic job entirely.
11 of 64 merges in the first eight weeks carried the
label. all eleven were correct to.The rollback, which was already there
switch_release "$new" && systemctl reload php-fpm
for i in $(seq 1 20); do
body=$(curl -sf localhost/health/deep) &&
[ "$(jq -r .commit <<<"$body")" = "$SHA" ] && ok=1 && break
sleep 2
done
[ -n "${ok:-}" ] || {
switch_release "$previous" && systemctl reload php-fpm
./bin/notify '#ops' "rolled back $SHA"
exit 1
}Comparing the deployed commit against the one being deployed is what makes the health check assert the right thing — a check served by a stale process passes otherwise, which is the failure this exists to catch. Forty seconds of retrying and a rollback that is the same function called with the previous release.
Verifying it worked
# three weeks after
merges 52
automatic deploys 41
manual (labelled) 11
median merge-to-deploy 6m 20s # was 1h 40m
batched merges 0
rollbacks 1
# the rollback: a health check failure caused by a
# config cache that was not rebuilt. reverted in 38
# seconds, unattended, at 14:02 on a Wednesday.
$ ./bin/incident-log --since=2025-04 --tag=deploy | wc -l
0Zero batched merges is the outcome, and the single rollback is the mechanism working — a failure attributable to one change, reverted before anybody looked at it. Six minutes from merge to production is a consequence of the ninety-second suite rather than of anything in this change.
What this costs
A merge that is now irreversible in a way a merge was not. Reviewing a pull request is now reviewing a deployment, and the pressure that used to sit on the person running the deploy has moved to the person clicking merge — which is the right place for it and is a change in what approval means.
The migration check is also a pattern match on SQL, which is a weak tool. It will pass something dangerous eventually, and the marker convention depends on an author making an accurate claim about which release stopped reading a column — a claim nobody verifies except by reading the deployment history, which nobody does.