A question arrived about a release from July: did it contain a particular version of a library. The answer should have been a rebuild and a check, and the rebuild produced a different image from the one that had shipped — with no way to tell which parts of the difference mattered.
The symptom
$ git checkout v4.11.0 # tagged 2023-07-18
$ docker build -t app:rebuild .
$ docker images --format '{{.Repository}}:{{.Tag}} {{.Size}}'
app:rebuild 212MB
app:v4.11.0 199MB
$ docker run --rm app:v4.11.0 composer show | md5sum
4a7e8c1f...
$ docker run --rm app:rebuild composer show | md5sum
9b2d0e6a...
# same commit. same Dockerfile. different everything.Thirteen megabytes and a different dependency listing from an identical commit. Neither image is wrong — both are what the Dockerfile describes — and there is no version of “what shipped in July” that can be recovered from the repository.
Why it happens
A Dockerfile is a recipe rather than a specification. It names what to install and mostly does not name which version, so the result depends on when it is run — and every layer of the stack has a different way of not being pinned.
The fix
Four sources of non-determinism, in order of size
1 the base image FROM php:8.3-fpm-bookworm
a tag is a moving pointer. between July and
January it moved eleven times: PHP 8.3.0 to
8.3.2, and a Debian point release.
2 apt packages no lock file exists.
apt-get install libicu-dev installs whatever the
mirror currently has. 41 packages, none pinned.
3 composer composer.lock is honoured,
and platform requirements are resolved against
the PHP that happens to be present.
4 timestamps and
file ordering every layer hash includes
mtime. a checkout sets mtime to now.The fourth is the one that makes bit-identical rebuilds hard and the first three are the ones that make the *contents* differ. They need different treatment, and conflating them is why this problem gets abandoned — a bit-identical rebuild is a research project, and knowing what was in an image is an afternoon.
Digest pinning, which is the easy half
# not this
FROM php:8.3-fpm-bookworm
# this
FROM php:8.3-fpm-bookworm@sha256:a4c8f9e2b1d7c3e5f8a0b2d4c6e8f0a2b4d6c8e0f2a4b6d8c0e2f4a6b8d0c2e4
# and the base image's own build date, recorded, because
# the digest is opaque
# base built: 2024-01-08, PHP 8.3.2, Debian 12.4A digest is immutable and it is also unreadable, so the comment carrying the version it corresponds to is not decoration — it is the only way a human reviewing a bump knows what changed. The bot that opens the bump pull request writes that comment from the image labels.
The apt problem, costed and not solved
the options, all of which were considered:
pin every version explicitly
apt-get install libicu72=72.1-3
→ breaks the moment a security update ships, which
is exactly when you want it to not break.
a snapshot mirror (snapshot.debian.org)
→ genuinely solves it. adds a dependency on an
external service for every build, and it is slow.
host our own mirror snapshot
→ 40 GB and a job. costed at two days plus ongoing.
record what was installed, and do not pin
→ does not give reproducibility. gives the answer
to the question actually being asked.
we chose the last one.This is the decision worth defending: the question was “what was in the July image”, not “can we rebuild the July image byte for byte”. Recording is a hundred times cheaper than reproducing and answers the question completely, and the security posture — an unpinned apt gets security updates — is better rather than worse.
The build manifest
# generated inside the image at build time
{
echo "commit: $(git rev-parse HEAD)"
echo "built_at: $SOURCE_DATE_EPOCH"
echo "base_digest: $BASE_DIGEST"
echo "php: $(php -r 'echo PHP_VERSION;')"
echo '---'
dpkg-query -W -f='${Package}=${Version}n' | sort
echo '---'
composer show --format=json | jq -r
'.installed[] | "(.name)=(.version)"' | sort
} > /app/BUILD_MANIFEST$ docker run --rm app:v4.11.0 cat /app/BUILD_MANIFEST | head -6
commit: 8c1f4a7e9b2d0e6a3f5c7b9d1e3a5c7b9d1e3a5c
built_at: 1689696000
base_digest: sha256:a4c8f9e2...
php: 8.3.0
---
adduser=3.134
# and the question that started this:
$ docker run --rm app:v4.11.0 cat /app/BUILD_MANIFEST
| grep -c 'vendor/affected-lib=2.4.1'
1The manifest is inside the image, which means it travels with it and cannot be lost — an external record in a database is a second thing that can be wrong. It is also published as a build artefact, so the question can be answered without pulling a two-hundred-megabyte image.
SOURCE_DATE_EPOCH, and what it does not fix
- name: build
env:
SOURCE_DATE_EPOCH: ${{ github.event.head_commit.timestamp }}
run: |
docker buildx build
--build-arg SOURCE_DATE_EPOCH
--output type=image,rewrite-timestamp=true
-t app:${{ github.sha }} .with timestamps normalised, two builds of the same
commit on the same day:
layer 1 (base) identical
layer 2 (apt) DIFFERENT — a package moved
layer 3 (extensions) identical
layer 4 (vendor) identical
layer 5 (source) identical
four of five identical, and the one that differs is
apt, which we deliberately did not pin.
which is the correct outcome and is not "reproducible".Normalising timestamps removed the noise and left one genuine difference, which is what makes the result readable — before this every layer differed and there was no way to see that only one of them mattered. Four of five identical is a diagnostic tool rather than a guarantee.
Where the manifest lives, and who reads it
three copies, deliberately:
in the image /app/BUILD_MANIFEST. travels with
the artefact and cannot be
separated from it.
as a CI artefact attached to the release, so the
question can be answered without
pulling 200 MB.
in the release
notes a link, and the base image version
in plain text, because the person
asking in five years will start
from the changelog.
the third one is the only one anybody has used.Nobody has ever pulled the image to read the manifest, and the release notes link has been followed four times — which says something about how these questions actually arrive. They come from somebody reading a changelog and asking whether a version was affected, not from an engineer with a shell open.
The composer half, which was already nearly right
# the lock file pins the versions. what it does not pin:
# the PHP version they were resolved against
# the platform extensions available
# so: declare the platform, and check it
$ composer config platform.php 8.3.0
$ composer check-platform-reqs --no-dev
# and in CI, the assertion that the lock is honoured
$ composer install --no-dev --dry-run 2>&1 | grep -c 'Updating'
0Rebuilding three historical releases
# with digest pinning applied retroactively, from the
# manifests we had started keeping in October
$ ./bin/rebuild-release v4.19.0 # 2023-11
vendor: identical
apt: 3 packages differ (security updates)
php version: identical
VERDICT: contents accounted for
$ ./bin/rebuild-release v4.22.0 # 2023-12
VERDICT: contents accounted for
$ ./bin/rebuild-release v4.11.0 # 2023-07
no manifest. no base digest recorded.
VERDICT: cannot be determinedThe July release remains unanswerable, which is the honest outcome and was the whole reason for the exercise. Everything from October onward can be accounted for, and the gap is permanent — there is no retroactive fix for a build that did not record what it used.
Verifying it worked
$ ./bin/verify-manifest app:latest
manifest present: yes
commit matches tag: yes
base digest pinned: yes
composer platform set: yes
apt packages recorded: 412
# a deliberate check: build the same commit twice
$ ./bin/diff-layers app:build-1 app:build-2
layers identical: 4 of 5
differing: the apt layer (expected)
differing packages: 0 # same day, same mirror
# and the CI gate
$ ./bin/verify-manifest app:${SHA} || exit 1What this costs
A manifest that must be published with every release, forever, and a verification step in the pipeline that will one day fail for a reason unrelated to anything anybody cares about. The manifest is also four hundred lines of text per release, which is nothing in storage and is a thing to keep somewhere findable in five years.
The base image digest also has to be bumped deliberately, which means a security update to Debian now requires a pull request rather than arriving on the next build. That is the trade — the previous arrangement got security updates automatically and could not tell you what it had — and it means the bump bot is now load-bearing rather than a convenience.
The largest cost is a habit. Recording rather than reproducing is the right answer for this question and it is not the right answer for a supply-chain attestation, where somebody genuinely needs to verify that an artefact corresponds to a source tree. If that requirement arrives, this work is a foundation and not a solution, and the apt decision would have to be revisited.