The Dockerfile that finally stopped installing the world

The production image was 1.4 gigabytes and took six minutes to build, for an application whose source is forty megabytes. Nothing about it was wrong in a way that anybody could point at — it had simply been added to for four years, and every addition had been reasonable at the time.

The symptom

$ docker history app:latest --format '{{.Size}}t{{.CreatedBy}}' | head -8
612MB   RUN apt-get install build-essential autoconf...
318MB   RUN pecl install redis imagick xdebug
188MB   RUN apt-get install nodejs npm
142MB   COPY . /app
 91MB   RUN composer install
 44MB   RUN apt-get install libicu-dev libzip-dev

$ docker run --rm app:latest which gcc make node npm
/usr/bin/gcc
/usr/bin/make
/usr/bin/node
/usr/bin/npm

A compiler, a build system and a JavaScript runtime in a production image that runs PHP. None of them are used after the build and all of them are attack surface, and the 612 megabytes of build tooling is the single largest layer.

Why it happens

A single-stage Dockerfile has nowhere to put the things that are needed to build but not to run. Every extension that has to be compiled brings a toolchain, and the toolchain stays because removing it in a later RUN does not shrink the layer that added it.

The fix

A build stage that is allowed to be enormous

FROM php:8.2-fpm-bookworm AS build

RUN apt-get update && apt-get install -y --no-install-recommends 
      build-essential autoconf libicu-dev libzip-dev libpng-dev 
 && docker-php-ext-install -j"$(nproc)" intl zip gd pdo_mysql opcache 
 && pecl install redis && docker-php-ext-enable redis

# this stage is 900 MB and never leaves the builder
FROM php:8.2-fpm-bookworm AS runtime

# the shared libraries the extensions link against — and
# nothing that compiles anything
RUN apt-get update && apt-get install -y --no-install-recommends 
      libicu72 libzip4 libpng16-16 
 && rm -rf /var/lib/apt/lists/*

COPY --from=build /usr/local/lib/php/extensions/ /usr/local/lib/php/extensions/
COPY --from=build /usr/local/etc/php/conf.d/ /usr/local/etc/php/conf.d/

Copying the compiled .so files rather than rebuilding is what makes this work, and it requires the runtime libraries to be present — libicu72 rather than libicu-dev. Getting that list right is the fiddly part and the failure is loud: the extension fails to load at startup and names the missing library.

The cache mount, and where it does not help

FROM build AS vendor
WORKDIR /app
COPY composer.json composer.lock ./
RUN --mount=type=cache,target=/tmp/composer 
    COMPOSER_CACHE_DIR=/tmp/composer 
    composer install --no-dev --no-scripts --no-autoloader --prefer-dist

COPY . .
RUN composer dump-autoload --classmap-authoritative --no-dev

Copying the lock file before the source is the ordering that makes the layer cache work — a change to a controller does not reinstall dependencies. The cache mount handles the other case, where the lock file did change and nine tenths of the packages did not, and it is worth measuring separately because on a hosted CI runner with a fresh machine it does nothing at all.

Ordering by change frequency

the rule: least-frequently-changed first.

  base image and system packages    monthly
  php extensions                    quarterly
  composer dependencies             weekly
  application source                many times a day

inverting any two of those costs a rebuild of everything
below. we had COPY . . above the composer install, which
meant every source change reinstalled 88 packages.

This is the whole optimisation and it is free. The five minutes we had been spending were almost entirely a single misordered COPY, and the multi-stage work — which is the part that sounds clever — bought the image size rather than the time.

What could not be removed

the final image, by layer:

  php:8.2-fpm-bookworm base        104 MB
  runtime shared libraries          38 MB
    of which libicu72               32 MB
  compiled extensions                4 MB
  vendor                            41 MB
  application source                12 MB
  ────────────────────────────────────────
                                   199 MB

ICU is 32 MB and is not optional: the application
formats currency and dates for four locales.

Thirty-two megabytes for internationalisation data is the largest remaining item and there is no version of this that removes it. Knowing that is worth more than the number — it stops the next person spending a day trying, and it is the reason the image will not get much below 190 megabytes.

Verifying it worked

$ docker images app --format '{{.Size}}'
199MB          # was 1.4GB

$ time docker build .            # cold, no cache
real    2m14.8s                  # was 6m02s

$ echo 'x' >> src/Http/Controller.php && time docker build .
real    0m18.2s                  # was 4m41s

$ docker run --rm app:latest sh -c 'which gcc make node'
# (no output)

$ docker run --rm app:latest php -m | tr 'n' ' '
Core ctype curl dom gd iconv intl json mbstring opcache
pdo pdo_mysql redis zip

The eighteen seconds on a source-only change is the number that changed how the day feels, and it comes entirely from the reordering. Asserting that the compilers are absent is worth doing in CI rather than by eye, because the next person adding an extension will reach for the build stage’s package list by habit.

What this costs

A Dockerfile that is harder to read than the one it replaced, with three stages and a list of runtime libraries that has to be kept in step with the extensions. Adding an extension is now a two-place change and the failure when you get it wrong is at container start rather than at build, which is later than it should be.

The runtime library list is also the part that will rot. It was derived by running ldd over the compiled extensions, which is a thing to write down in a comment rather than a thing to remember — and the comment is now the only documentation of why libpng16-16 is installed in an image that serves no images directly.