The container scanner had been reporting forty-odd vulnerabilities a week for two years and had never once produced an action. That is not a scanner problem — it is what happens when a general-purpose base image carries a general-purpose package set and the report cannot tell which packages your application actually loads.
The symptom
$ trivy image app:latest --severity HIGH,CRITICAL
Total: 41 (HIGH: 36, CRITICAL: 5)
CVE-2023-xxxx perl-base HIGH
CVE-2023-xxxx libperl5.36 HIGH
CVE-2023-xxxx git HIGH
CVE-2023-xxxx openssh-client CRITICAL
CVE-2023-xxxx libtiff6 HIGH
... 36 more
$ docker run --rm app:latest sh -c 'which perl git ssh'
/usr/bin/perl
/usr/bin/git
/usr/bin/sshPerl, git and an SSH client in an image that runs a PHP application behind a socket. None of them are invoked by anything, all of them are in the report every week, and their presence is why nobody reads it.
Why it happens
An official language image is built to be useful to everybody, so it includes the tooling most people will need. That is the right decision for the image and it means the vulnerability surface of your container is determined by other people’s requirements.
The fix
Building from slim, with what we load
FROM debian:12-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends
ca-certificates libicu72 libzip4 libpng16-16 libxml2
libsodium23 libcurl4 zlib1g
&& rm -rf /var/lib/apt/lists/*
COPY --from=php-build /usr/local/bin/php-fpm /usr/local/bin/
COPY --from=php-build /usr/local/lib/php/ /usr/local/lib/php/
COPY --from=php-build /usr/local/etc/php/ /usr/local/etc/php/
# no perl, no git, no ssh, no package manager after this pointCompiling PHP ourselves is the part that sounds like too much work and takes about forty lines, once. The list of runtime libraries is derived from ldd over the binary and the extensions, and getting it wrong produces a startup failure that names exactly what is missing.
The rebuild cadence
on:
schedule:
- cron: '0 4 * * 1' # weekly, Monday
workflow_dispatch:
jobs:
base:
steps:
- run: docker buildx build --push -t base:${{ github.run_number }} .
- run: trivy image --exit-code 1 --severity CRITICAL base:${{ github.run_number }}
- run: |
echo "BASE_DIGEST=$(crane digest base:${{ github.run_number }})"
>> base-digest.envWeekly rather than on-demand is deliberate: a base rebuilt only when somebody remembers is a base that is six months old when the vulnerability that matters is published. The scan failing the build on a critical finding is what makes the rebuild meaningful rather than a scheduled no-op.
Pinning downstream, and promoting deliberately
the application Dockerfile pins by digest:
FROM registry/base@sha256:a4c8f9e2...
and a bot opens a pull request when the base digest moves.
the pull request body carries:
packages changed: libssl3 3.0.9 → 3.0.11
libcurl4 7.88.1 → 7.88.1-10
CVEs resolved: CVE-2023-xxxx (HIGH)
scan result: 0 CRITICAL, 2 HIGH (both in
libsystemd, no fix available)
which is a review that takes ninety seconds and is
actually a review.The package diff in the pull request body is what turns an automated bump into a decision. Without it the bot is a rubber stamp with extra ceremony; with it, somebody reads two lines and knows whether this is routine or worth attention.
The scan, again
$ trivy image app:latest --severity HIGH,CRITICAL
Total: 3 (HIGH: 3, CRITICAL: 0)
CVE-2023-xxxx libsystemd0 HIGH no fix available
CVE-2023-xxxx libgcrypt20 HIGH no fix available
CVE-2023-xxxx libc6 HIGH fixed in 2.36-9+deb12u2
# the third one was actionable, and was actioned.
# the first two are tracked with a note and re-checked
# on every rebuild.Three findings, one of them actionable, is a report somebody reads. The two without fixes are the ones that need a note explaining why they are accepted — a base with no systemd would remove one of them and would mean a different init story, which was judged not worth it.
What this makes worse
before: the base image was somebody else's problem.
after: it is ours.
which means:
a PHP patch release is our upgrade to perform
a Debian point release is our rebuild
a broken build on a Monday morning is ours
the ldd-derived library list is ours to maintain
and the failure mode: the rebuild job is disabled
during an incident and nobody re-enables it. we added
a check that fails the application build if the base
is older than 21 days.The staleness check is the piece that makes this survivable, because the realistic failure is not a bad rebuild — it is no rebuild for four months. Failing the application build after three weeks is aggressive and it is the only thing that reliably produces a rebuild.
Verifying it worked
$ docker images --format '{{.Repository}} {{.Size}}'
app 168MB # was 199MB
base 96MB
$ trivy image app:latest --severity HIGH,CRITICAL | tail -1
Total: 3 (HIGH: 3, CRITICAL: 0) # was 41
$ docker run --rm app:latest php -m | wc -l
17 # unchanged
$ docker run --rm app:latest sh -c 'apt-get --version'
sh: apt-get: not found
# and a deliberate test of the staleness check
$ BASE_BUILT_AT=2023-01-01 make build
base image is 62 days old (max 21). run the base workflow.The absent package manager is worth asserting deliberately — it is the difference between a hardened image and one that merely happens not to have anything installed. Testing the staleness check by lying about the build date confirms the guard works before it is needed.
What this costs
A second artefact with its own pipeline, its own scan and its own way of breaking, maintained by a team of four. The weekly rebuild is unattended until it fails, and it will fail on a Monday when a Debian package rename breaks the install line.
The library list derived from ldd is the part that will rot quietly. Adding a PHP extension means adding its shared libraries to the runtime stage, and forgetting produces a container that starts fine and fails when that extension is first used — which may be weeks later, on one code path.
The honest summary is that this converts a noisy report into a quiet one and takes on real maintenance to do it. For a team with no security requirement beyond diligence, the alternative — staying on the official image and accepting that the report is noise — is defensible, and it is what we had been doing for two years.