Seven services in one development environment

Onboarding took two days and produced a machine that was almost right. The wiki page describing the setup had been edited eleven times and was still wrong about the PHP version, because the person who upgraded it had upgraded their own machine and not the page. This follows from the per-service images — the Dockerfiles are the components, and this is the assembly.

The symptom

# the wiki page, abridged
1. brew install [email protected]        ← project is on 7.2
2. brew install [email protected] redis && pecl install redis
3. copy .env.example, edit 14 values
4. ask someone for the API keys
5. mysql -u root < dump.sql   (ask someone which dump)
6. if step 2 fails see the note at the bottom, from 2016

Every step is somebody’s afternoon and step 6 is somebody else’s. The deeper problem is that no two machines end up identical, so “works on my machine” is a statement about a configuration nobody can reproduce.

Why it happens

Every service was installed by hand, in a different order, by a different person, over three years. Nothing recorded the result except a wiki page maintained by whoever was most recently annoyed by it. There is no mechanism in that arrangement that converges — each machine drifts independently and the drift is invisible until it causes a bug that only one person can reproduce.

The fix

The compose file, and networks that keep it legible

version: '3.7'

x-app: &app
  restart: unless-stopped
  networks: [app]
  env_file: [.env]
  logging:
    driver: json-file
    options: { max-size: '10m', max-file: '3' }

services:
  nginx:
    <<: *app
    build: { context: ., dockerfile: docker/nginx/Dockerfile }
    ports: ['8080:80']

  php:
    <<: *app
    build: { context: ., dockerfile: docker/php/Dockerfile }
    volumes: ['./:/app', 'vendor:/app/vendor']

  mysql:
    <<: *app
    image: mariadb:10.3
    volumes: ['mysql:/var/lib/mysql', './docker/mysql:/docker-entrypoint-initdb.d:ro']

  redis: { <<: *app, image: 'redis:5-alpine' }
  mail:  { <<: *app, image: 'mailhog/mailhog', ports: ['8025:8025'] }

  queue:
    <<: *app
    image: schickling/beanstalkd
    command: ['-b', '/data', '-f', '1000']
    volumes: ['beanstalk:/data']

networks: { app: {} }
volumes: { mysql: {}, vendor: {}, beanstalk: {} }

The x-app anchor is the thing that keeps this readable at seven services. Any top-level key beginning x- is ignored by compose and available as a YAML anchor, so the shared restart policy, network and log rotation are declared once. The log options are not decoration — the default json-file driver has no size limit, and a chatty container will quietly consume the host disk over a fortnight.

One network rather than several is deliberate. Segmenting a development stack into front and back networks looks tidy and produces an afternoon of debugging the first time somebody needs the queue worker to reach Redis. Production can be segmented; the laptop does not need to be.

The mail catcher, and the accident it prevents

A staging database restored from production contains real addresses, and a queue worker running against it sends real email to real customers. It happens to everybody once.

# .env.example — committed, so this is the DEFAULT
MAIL_DRIVER=smtp
MAIL_HOST=mail
MAIL_PORT=1025

# and the belt to that brace, on every non-production host:
$ ufw deny out 25/tcp

Putting the catcher in the example environment file rather than in a wiki page is what makes this reliable, because the failure mode is somebody copying a stale .env. It also makes email testable — the web interface shows the raw message, so an encoding problem in a header is visible rather than reported by a customer.

Volumes: what persists, what must not, and the slow one

volumes:
  - ./:/app                  # bind mount: the source. slow on macOS.
  - vendor:/app/vendor       # named volume: seeded from the image
  - /app/storage/framework   # anonymous: discarded, and fast

# the named volume is seeded ONCE. after a dependency change:
#   docker-compose down -v
# or the container runs last week's packages.

The named volume for vendor is the trick everyone on macOS eventually adopts, and it has a failure mode that is genuinely confusing: the volume is seeded once from the image, so a composer require on the host does not appear in the container. Documenting the removal command next to the compose file saves the same twenty minutes repeatedly.

The database volume is the one to be careful with. docker-compose down -v deletes it, which is fine for a stack seeded from a script and is a bad afternoon if it held the only copy of something. Seeding from docker-entrypoint-initdb.d makes the data reproducible, which is what makes deleting it safe.

Starting in a predictable state without depending on order

#!/usr/bin/env sh
set -eu

until nc -z "${DB_HOST:-mysql}" 3306; do sleep 1; done

[ -f /app/vendor/autoload.php ] || composer install --no-interaction

exec "$@"

Waiting in the entrypoint rather than relying on depends_on is the arrangement that works, because compose can promise start order and cannot promise readiness — and in a v3 file the condition form of depends_on is ignored entirely, silently. The exec at the end matters: without it the shell stays as PID 1 and signals never reach the real process, so docker-compose stop takes ten seconds and then kills it.

Installing dependencies on first boot is convenient and is the line most likely to be argued about. It makes the first start slow and every subsequent one instant, and it means a clone-and-run works with no host toolchain at all — which is the property the whole exercise is for.

The Makefile, which is the actual interface

setup:
	printf 'UID=%snGID=%sn' "$(shell id -u)" "$(shell id -g)" > .env.docker
	cp -n .env.example .env || true
	docker-compose build

up:    ; docker-compose up -d && docker-compose ps
shell: ; docker-compose exec php sh
test:  ; docker-compose exec -T php vendor/bin/phpunit
fresh: ; docker-compose down -v && $(MAKE) up

Nobody should have to know the compose commands. Six make targets is the whole interface a new person needs, and it is the thing that replaces the wiki page — because unlike a wiki page it stops working when it is wrong, which is the only reliable form of documentation.

Verifying it worked

# on a laptop that has never seen this project
$ git clone git@internal:shop/app.git && cd app
$ make setup && make up && open http://localhost:8080

real    4m12s      # first run, cold image pull
real    0m18s      # every subsequent make up

$ docker stats --no-stream --format '{{.Name}}t{{.MemUsage}}'
app_mysql_1    482MiB    app_php_1    212MiB
app_nginx_1     12MiB    app_redis_1    9MiB
# total, seven services: ~1.1 GiB

The clean-machine test is the only verification worth anything, and doing it on somebody else’s laptop rather than your own is what catches the step that lives in your shell history. Two days to four minutes is the number that justified the work; the eighteen seconds afterwards is the number that changed how often people tear the stack down.

What this costs

About a gigabyte of RAM and a laptop fan that never stops. On an eight-gigabyte machine running an editor and a browser this is genuinely tight, and it is worth having a documented way to run a subset — the API and MySQL without the queue, the mail catcher or Apache — for the days when the whole stack is not needed. Compose can start named services individually, which is the answer, and nobody discovers it without being told.

The second cost is that a problem in the stack is now a problem nobody can fix by knowing PHP. The failure modes are DNS between containers, a volume with stale contents, a port already bound, and an image that needs rebuilding — a small operational skill set that every developer now needs a little of. That is a real tax, and it is smaller than the two days it replaced.