The integration suite passed locally and could not run in the pipeline. Locally there was a compose file with seven services; in CI there was an Ubuntu machine with nothing on it. Bridging those two turned out to be a set of small decisions, each of which has a wrong answer that works until it does not.
The symptom
$ vendor/bin/phpunit --testsuite integration
SQLSTATE[HY000] [2002] Connection refused
$ docker-compose ps
ERROR: Couldn't connect to Docker daemon
# and the workaround that had been shipped, in phpunit.xml:
<testsuite name="integration">
<!-- disabled in CI, see ENG-2841 -->
</testsuite>The integration suite had been commented out of CI for four months, which means the tests that check what the database actually does had been running only on developer machines — where they pass, because that is where they were written.
Why it happens
The local stack is a compose file and the pipeline is not, so there are two descriptions of the same set of services and the second one does not exist. The obvious fix is to run the compose file in CI, and the obvious fix is slower and more fragile than the alternative — which is why this is a decision rather than a lookup.
The fix
Service containers, and where they differ from compose
jobs:
integration:
runs-on: ubuntu-20.04
services:
mysql:
image: mysql:8.0
env: { MYSQL_ROOT_PASSWORD: root, MYSQL_DATABASE: app_test }
ports: ['3306:3306']
options: >-
--health-cmd="mysqladmin ping" --health-interval=10s
--health-timeout=5s --health-retries=5
redis:
image: redis:6-alpine
ports: ['6379:6379']
options: --health-cmd="redis-cli ping" --health-interval=10scompose service containers
---------------------------------------------------------
services reach each other by the STEPS run on the host;
name, on a private network services publish ports to it
→ connect to 127.0.0.1
command and entrypoint can be only image, env, ports,
set per service volumes and docker options
depends_on with conditions nothing. health options only.
build: from a Dockerfile image only. no building.The host-versus-network difference is the one that catches everyone: the application connects to 127.0.0.1:3306 rather than to mysql:3306, because the steps are not containers. A test suite reading the host from an environment variable handles both; one with the hostname in a config file does not.
The inability to set a command is the limitation that decides some cases outright. A service needing arguments — Beanstalkd with its binlog flags, a Redis started with a custom configuration — cannot be a service container in 2020, which is a real constraint rather than an inconvenience.
Waiting for readiness, which nothing does for you
# the health options make the RUNNER wait — but only for the
# container's own healthcheck, and mysqladmin ping succeeds
# before MySQL will accept a connection on a fresh volume.
- name: Wait for MySQL
run: |
for i in $(seq 1 30); do
if mysqladmin ping -h 127.0.0.1 --silent; then exit 0; fi
sleep 1
done
echo 'mysql did not become ready' >&2; exit 1The health options genuinely help and are not sufficient on their own: mysqladmin ping returns success once the socket answers, and MySQL 8 spends several more seconds initialising the data directory on a cold start. The failure is an intermittent connection refused on about one run in fifteen, which is exactly the flakiness that makes people re-run pipelines.
An explicit wait step with a bounded loop and a clear failure message is ten lines and removes the class entirely. Failing loudly after thirty seconds is better than a test failure that gets attributed to whatever changed most recently.
When to use the compose file instead
- name: Bring up the stack
run: |
docker-compose -f docker-compose.yml -f docker-compose.ci.yml
up -d --no-build
docker-compose exec -T php ./wait-for-services.sh
- run: docker-compose exec -T php vendor/bin/phpunit
- if: failure()
run: docker-compose logs --no-color --tail=200service containers faster to start (~15s), simpler, and
a SECOND description of the stack
compose in CI the SAME description as local, can build,
can set commands — and ~70s to start
the deciding question: how likely is the second description
to drift from the first, and how much does that cost?Duplication is the real cost of service containers, and it is a duplication of infrastructure definitions rather than of code — so it drifts silently and is discovered when a version differs. Running the compose file means one description and a slower pipeline, which for a suite that runs on every push is a genuine trade.
The arrangement that settled was service containers for the fast unit and integration jobs and the compose file for a single end-to-end job that runs on the default branch only. Two descriptions, and the expensive one exercises the same file developers use.
The if: failure() log step is worth having in either arrangement. A container that failed to start produces no output at all otherwise, and the pipeline reports a connection error with no indication of why.
Caching layers so the pipeline is not slower than the tests
- uses: docker/setup-buildx-action@v1
- uses: docker/build-push-action@v2
with:
context: .
file: docker/php/Dockerfile
load: true
tags: app-php:ci
cache-from: type=gha
cache-to: type=gha,mode=maxWithout a cache, building the PHP image on every run is two minutes for a suite that takes forty seconds — and building is what a compose-based pipeline does unless --no-build is passed. The GitHub Actions cache backend for BuildKit is new in 2020 and is the piece that makes the compose route affordable at all.
mode=max caches intermediate layers rather than only the final image, which is what helps a multi-stage build. It also stores considerably more, and the cache has a size limit that evicts least-recently-used entries — so a repository with several images can find them evicting each other.
Verifying it worked
$ php artisan test --testsuite=integration
Tests: 96 passed
# in CI, twenty consecutive runs
# passed: 20
# flaky: 0 (was ~1 in 15 before the wait step)
# median: 1m 21s
# and the check that the two descriptions agree
$ diff
<(grep -oP 'image: KS+' docker-compose.yml | sort)
<(grep -oP 'image: KS+' .github/workflows/ci.yml | sort)
< mailhog/mailhog
< nginx:1.19-alpine
# expected: CI does not need those twoDiffing the image lists between the compose file and the workflow is the cheapest guard against the duplication drifting, and it belongs in CI rather than in somebody’s memory. The two legitimate differences are worth listing in the check itself so that a third one fails the build.
Twenty consecutive green runs is the assertion about flakiness, and it is worth doing deliberately after adding the wait step — one green run proves nothing about a race that happens one time in fifteen.
What this costs
Two descriptions of the same stack, kept in step by hand. The image-list diff catches version drift and catches nothing about configuration: a compose file that sets a MySQL character set and a workflow that does not produces a suite passing locally and failing in CI for a reason nobody connects to the difference. That is the failure this arrangement makes possible and it is the price of the faster pipeline.
The alternative — one description, running compose everywhere — costs about a minute per run, which on a repository with forty pushes a day is forty minutes of somebody waiting. Neither answer is clearly right, and the useful thing is to know which cost you are choosing rather than to discover it.