A deployment pipeline that runs the tests first

The deploy was a shell script one person had on their laptop. It ran rsync, it worked, and it had never been reviewed by anyone. The release that finally forced the issue passed locally, passed in review, and took the checkout page down for eleven minutes because a migration had not been run on the server.

The symptom

#!/bin/bash
rsync -az --delete ./ deploy@web:/var/www/shop/
ssh deploy@web 'cd /var/www/shop && php artisan migrate --force'
ssh deploy@web 'sudo systemctl reload php7.0-fpm'

Four problems in four lines. --delete against the live directory means the site is inconsistent for the duration of the copy. The migration runs after the code, so for those seconds the new code is querying the old schema. Nothing verifies the tests passed. And there is no way back except running an older checkout through the same script.

Why it happens

Nothing between the commit and the server disagrees with the person deploying. The tests exist and running them is optional; the migration is remembered rather than enforced; the state of the server is whatever the last few deploys left behind. Each individual step is fine and the sequence has no gate in it.

The fix

Stages, in the order that fails cheapest first

# .gitlab-ci.yml
stages: [lint, test, build, stage, production]

lint:
  stage: lint
  script:
    - composer validate --strict
    - vendor/bin/phpcs --standard=phpcs.xml src

test:
  stage: test
  services: [mysql:5.7, redis:3.2]
  script:
    - composer install --prefer-dist --no-interaction
    - php artisan migrate --force
    - vendor/bin/phpunit --colors=never

build:
  stage: build
  script:
    - composer install --prefer-dist --no-dev --optimize-autoloader
    - npm ci && npm run build
    - tar -czf release.tar.gz --exclude=.git .
  artifacts:
    paths: [release.tar.gz]
    expire_in: 1 week

Lint before test because it takes eight seconds and catches a whole class of mistake. Test before build because there is no point compiling assets for a release that will not ship. Build once and carry the artefact forward, so staging and production receive byte-identical code rather than two independent builds that could differ.

Atomic releases

The copy-into-place problem is solved the same way it has been for fifteen years: build the new release beside the old one and move a symlink.

/var/www/shop/
├── releases/
│   ├── 20160908141122/
│   ├── 20160912093401/
│   └── 20160913160244/     <- just extracted
├── shared/
│   ├── .env
│   └── storage/
└── current -> releases/20160912093401

set -euo pipefail
REL=/var/www/shop/releases/$(date +%Y%m%d%H%M%S)

mkdir -p "$REL" && tar -xzf release.tar.gz -C "$REL"

ln -sfn /var/www/shop/shared/.env     "$REL/.env"
ln -sfn /var/www/shop/shared/storage  "$REL/storage"

php "$REL/artisan" migrate --force
php "$REL/artisan" config:cache

# atomic: rename over the symlink, never ln -sfn onto it
ln -sfn "$REL" /var/www/shop/current.tmp
mv -Tf /var/www/shop/current.tmp /var/www/shop/current

sudo systemctl reload php7.0-fpm

ls -1dt /var/www/shop/releases/* | tail -n +6 | xargs rm -rf

The ln then mv -T is the part that matters. ln -sfn directly onto an existing symlink unlinks and recreates, so there is a window — small, real, and long enough to serve a 404 — where current does not exist. A rename over it is atomic at the filesystem level.

Migrations run before the swap, against the new code, while the old code is still serving. That constrains what a migration may do: it must be compatible with both versions. Adding a column is fine, dropping one is a two-release operation, and that discipline is the price of not having downtime.

The rollback is the same mechanism

PREV=$(ls -1dt /var/www/shop/releases/* | sed -n 2p)

ln -sfn "$PREV" /var/www/shop/current.tmp
mv -Tf /var/www/shop/current.tmp /var/www/shop/current
sudo systemctl reload php7.0-fpm

Three lines, and they are the deploy’s three lines with a different directory. A rollback path that is a special case is a rollback path nobody has tested — this one runs the same code as every deploy, so it works because the deploy works.

What it does not roll back is the database, and that is not solvable by tooling. The two-release rule for destructive migrations is what makes the code rollback sufficient on its own.

Promotion, not redeployment

stage:
  stage: stage
  script: [./bin/deploy staging release.tar.gz]
  environment: { name: staging, url: 'https://staging.example.com' }

production:
  stage: production
  script: [./bin/deploy production release.tar.gz]
  environment: { name: production, url: 'https://shop.example.com' }
  when: manual
  only: [master]

when: manual makes production a button rather than a consequence of merging, and the artefact is the same file that was tested and staged. That is the difference between promotion and redeployment: nothing is rebuilt, so nothing can differ.

Verifying it worked

The test that matters is a deliberately broken commit, taken all the way to the gate it should not pass.

$ git commit -m 'deliberately break a test' && git push

lint       passed   00:09
test       FAILED   01:47
  1) OrderTest::testTotalIncludesVat
     Failed asserting that 4900 is identical to 5782.
build      skipped
stage      skipped
production skipped

And a smoke check after each deploy, which is what turns a successful script into a successful release:

for i in $(seq 1 10); do
  code=$(curl -fsS -o /dev/null -w '%{http_code}' https://shop.example.com/health || true)
  [ "$code" = "200" ] && exit 0
  sleep 2
done

echo 'smoke check failed, rolling back' >&2
./bin/rollback production
exit 1

What the pipeline must not be allowed to do

A runner that can deploy to production is a machine that executes untrusted code and holds production credentials. Either fact is fine alone and the combination is not, because the pipeline configuration lives in the repository and a branch can change it.

production:
  stage: production
  script: [./bin/deploy production release.tar.gz]
  when: manual
  only: [master]              # never from a branch
  environment: { name: production }

# the deploy credential is marked protected, so it is not exposed
# to a job running on any other ref

The deploy key is read-only on the repository and can write only to the release directory. That leaves the deploy script itself as the entire attack surface, and it is forty lines — short enough that reviewing it is realistic.

# /etc/sudoers.d/deploy — exactly two things, and nothing else
deploy ALL=(root) NOPASSWD: /bin/systemctl reload php7.0-fpm
deploy ALL=(root) NOPASSWD: /bin/systemctl reload nginx

Two lines rather than the blanket NOPASSWD: ALL the hand-rolled script had needed and nobody had revisited. It costs a sudoers entry each time the deploy learns something new, and that friction is the point: every addition is a decision instead of a default.

Making it fast enough that nobody routes around it

A pipeline people wait eleven minutes for is a pipeline people find ways around on a Friday. Most of that time was dependency installation repeated on every job, which is cacheable.

cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths: [vendor/, node_modules/]

test:
  stage: test
  script:
    - composer install --prefer-dist --no-interaction --no-progress
    - vendor/bin/phpunit

Keying the cache on the branch rather than globally avoids two branches with different lock files fighting over one cache, which produces the worst failure available here — a green build against dependencies that are not the ones in the lock file.

# before caching
lint 00:09   test 06:41   build 03:52   total 10:42

# after
lint 00:09   test 02:14   build 01:08   total 03:31

Three and a half minutes is short enough that nobody argues with it, which is worth more than the machine time saved. The one discipline it needs: a scheduled job that runs weekly with the cache disabled, or a broken lock file can sit behind a warm cache for a month without anybody finding out.

Keeping the pipeline honest about the environment

A pipeline that tests against a different stack from the one it deploys to is testing something else. The service containers are the cheapest place this drifts, because they are pinned in a file nobody reads after the day it is written.

test:
  image: php:7.0-cli          # same minor as production
  services:
    - mysql:5.7               # same as production
    - redis:3.2               # same as production
  variables:
    MYSQL_ROOT_PASSWORD: root
    DB_HOST: mysql
    REDIS_HOST: redis

And a check that says so out loud rather than relying on the file being correct:

$ ./bin/check-parity
php        ci 7.0.12   prod 7.0.12   ok
mysql      ci 5.7.16   prod 5.7.16   ok
redis      ci 3.2.5    prod 3.2.4    WARN minor drift
extensions ci 34       prod 34       ok

It reads the production versions from the deploy target over SSH and compares them with what the runner has. A warning rather than a failure, because a patch difference is normal and a major one is a conversation — the point is that somebody sees it before it becomes an incident with a confusing cause.

What this costs

The pipeline is now a system that can break, and it breaks in ways the old script could not — a runner out of disk, an artefact expiring before promotion, a service container that will not start. It needs the same care as production, and the temptation when it fails at five on a Friday is to deploy by hand, which undoes the entire point.

It also makes deploys slower. Eleven minutes from merge to production, against forty seconds of rsync. That trade is obviously right and it does not feel right on the day of a one-character copy fix, so it is worth agreeing in advance that there is no fast path — because the fast path is how the eleven-minute outage happened.

And the two-release rule for destructive migrations is a permanent constraint on how schema changes are written. Nobody enjoys splitting a column rename across two deploys, and every alternative involves downtime.