On the ninth of December a remote code execution vulnerability in a Java logging library became public. We do not write Java. The question arrived at 21:40 from somebody who does not care what language anything is written in, and it took until eight the next morning to answer with enough confidence to write it down.
The symptom
21:40 "are we affected by log4shell"
21:45 "we don't use Java" — confident, and not an answer
22:10 Elasticsearch is Java. (we run it)
22:30 Logstash is Java. (we ran it until 2020)
23:15 the CI runners have a JDK. (for one Android job)
00:40 a vendor's on-premise appliance. nobody knows.
02:00 three base images with a JRE nobody had noticed
08:10 a written answer
eleven hours. and "we don't use Java" had been wrong at
21:45 in four separate ways.Every one of those discoveries came from somebody remembering something rather than from querying anything. That is the actual failure — not that we were exposed, but that establishing whether we were took a night and depended on who happened to be awake.
Why it happens
An inventory maintained by hand is out of date the day it is written, so nobody maintains one. The information exists — in lock files, in image manifests, in base image package lists — and is scattered across places that are each authoritative for one layer and blind to the others.
The question is also not “do we use this package” but “does anything we run contain this, at a version below the fix”, which is a transitive question about deployed artefacts rather than about source repositories.
The fix
Generating it in the pipeline
# per build, alongside the image, keyed on the DIGEST
digest=$(docker image inspect "app:$TAG" -f '{{index .RepoDigests 0}}')
out="sbom/${digest##*@}"
mkdir -p "$out"
# 1. application dependencies, with transitives
composer show --format=json --tree > "$out/composer.json"
npm ls --all --json > "$out/npm.json" || true
# 2. what the base image contains — the layer everybody
# forgets, and where a JRE hides
docker run --rm --entrypoint sh "app:$TAG" -c
'(apk info -v 2>/dev/null || dpkg-query -W -f "=\n")'
> "$out/os.txt"
# 3. the metadata that makes it answerable later
jq -n --arg d "$digest" --arg t "$TAG" --arg c "$GITHUB_SHA"
'{digest:$d, tag:$t, commit:$c}' > "$out/meta.json"Keying on the image digest rather than the tag is what makes this answerable later — a tag is a moving pointer and the question is always about a specific deployed artefact. The operating system package list is the part that gets skipped and is exactly where the three unnoticed JREs were.
npm ls --all failing on a peer dependency warning is why the || true is there, and it is the sort of thing that silently produces an empty file. Asserting the output is non-empty before uploading is worth two more lines.
Knowing what is actually deployed
# an inventory of built images is half the answer. the
# other half is which digests are RUNNING, where.
for env in production staging; do
./bin/deployed-digests "$env" | while read -r svc digest; do
jq -n --arg e "$env" --arg s "$svc" --arg d "$digest"
'{env:$e, service:$s, digest:$d, seen:now}'
done
done > deployments.ndjson
# recorded on every deploy, appended, never overwritten —
# so "what was running on 9 December" is answerable tooRecording deployments as an append-only log rather than a current-state file is what lets the question be asked about a past moment, which matters when an advisory describes something that was fixed a fortnight ago. It is a few kilobytes a day and it turns “we think we upgraded that” into a query.
The query, which must take minutes
#!/usr/bin/env bash
# bin/sbom-query PACKAGE [--below VERSION]
set -euo pipefail
pkg="$1"; below="${3:-}"
jq -r --arg p "$pkg" '
.installed[] | select(.name == $p) | "(.name) (.version)"
' sbom/*/composer.json 2>/dev/null
grep -h -i "$pkg" sbom/*/os.txt || true
# usage, on the night:
# ./bin/sbom-query log4j
# ./bin/sbom-query 'openjdk*'$ ./bin/sbom-query 'log4j' --below 2.15.0
No match in 41 images across 2 environments.
$ ./bin/sbom-query 'openjdk'
production elasticsearch:7.10.2 openjdk-11-jre-headless=11.0.11
production ci-runner:2021-11 openjdk-11-jdk=11.0.11
staging elasticsearch:7.10.2 openjdk-11-jre-headless=11.0.11
# eleven hours the first time. four minutes the second.The version comparison is the part that has to work, because “do we use this” is rarely the question — “are we below the fixed version” is. Answering across environments matters too, since staging being patched says nothing about production.
What a scanner sees, and what it does not
$ trivy image app:2021-12-14
Total: 14 (UNKNOWN: 0, LOW: 8, MEDIUM: 2, HIGH: 3, CRITICAL: 1)
# what it found: apk and composer packages, from metadata
# what it did NOT find:
# a jar bundled inside another artefact
# a static binary fetched by a RUN curl | sh
# a vendored library COPYed in without a manifest
# anything mounted at runtime rather than built in
# a green scan means "nothing in the parts I can identify"The blind spots are consistent and deserve to be written next to the scan output, because a green result is otherwise read as “no vulnerabilities” rather than “none in the parts with package metadata”. Everything installed by a shell pipe in a Dockerfile is invisible to every scanner.
The other half is alert volume. A base image scan reports the same forty low-severity findings every day, and a policy that only fails the build on high severities with a fix available is what keeps anybody reading it — a scan nobody reads is worse than no scan, because it creates the impression of coverage.
The parts that stay manual
not covered by any of this, and listed explicitly so
nobody assumes otherwise:
a vendor's on-premise appliance — ask them, and record
the answer with a date
a managed service — their status page is the inventory
a browser extension a developer installed
anything in a lambda or a serverless function built
outside the pipeline
the CI runner image itself, if it is not ours
four of those five needed a person and an email. that is
fine, as long as the list of them exists.Writing down what the inventory does not cover is as valuable as the inventory, because it converts an unknown unknown into a known one with an owner. The vendor appliance took two days to get an answer about and would have taken two days regardless — the difference is that it was on a list rather than remembered at midnight.
Answering it to somebody who is not an engineer
The eleven hours were not spent finding out. They were spent getting from “we think we are fine” to a sentence somebody would put their name to, and that gap is a writing problem rather than a tooling one.
## CVE-2021-44228 — assessment, 10 Dec 08:10
**Are we affected:** No component we operate runs a
vulnerable log4j version.
**What we checked, and how**
| Layer | Method | Result |
|---|---|---|
| application deps | lock files, 41 images | no Java |
| base images | package lists, 41 images | 3 JREs, no log4j |
| deployed digests | release log, 2 envs | as above |
| managed services | vendor status pages | 2 confirmed |
| on-prem appliance | emailed vendor 09 Dec 23:10 | PENDING |
**What we cannot confirm:** the appliance. It is on an
isolated VLAN with no inbound internet. Escalated.
**Confidence:** high for everything we build, unknown for
one third-party appliance.Stating what was checked and how is what makes the answer auditable rather than reassuring, and listing the one thing that could not be confirmed is what makes the rest credible. An assessment with no unknowns in it is either a very simple estimate or an incomplete one.
The template being written in advance is the part worth copying. Composing this structure at two in the morning is why the last six hours took six hours — the facts were established by 02:00 and the document was not.
Rehearsing it
# quarterly, against a real advisory, with a stopwatch
$ ./bin/advisory-drill CVE-2021-44228
question posed 00:00
first query run 00:41
application deps 01:12 clear
base images 02:38 3 matches, JRE 11.0.11
deployed digests 03:50 2 environments
written answer 06:20
target: under 15 minutes. met.
# the drill also found the query script broken twice in
# a year, both times from a jq version change.The script rots exactly like everything else and the drill is the only thing that finds out. Twice in a year it was broken by an unrelated change, and both times that was discovered on a Tuesday afternoon rather than during an incident.
Verifying it worked
$ ls sbom/ | wc -l
412 # one per built image, keyed on digest
$ ./bin/sbom-query 'guzzlehttp/guzzle' --below 7.4.5
app-api 7.4.1 production deployed 2021-12-02
app-worker 7.4.1 production deployed 2021-12-02
app-admin 7.4.5 staging
$ ./bin/sbom-coverage
images built this quarter: 412
with an sbom: 412 (100%)
deployments recorded: 188
with a matching sbom: 188 (100%)The coverage check is the assertion that the inventory is complete, and it is the thing that will drift — an image built by a workflow somebody added last month without the sbom step is a gap that nothing else reports. Failing the deploy when a digest has no inventory is the stricter version and is worth the friction.
What this costs
An artefact per build and a scanner that cries wolf, plus a query script that is now infrastructure and needs the same care as anything else. The storage is trivial and the maintenance is not — three files per image, a format that changes when a tool updates, and a jq expression that nobody remembers writing.
The deeper cost is that it creates a false sense of completeness. The inventory covers the layers it can see, and the honest framing when reporting to anybody outside engineering is “we can answer this in minutes for everything we build, and there are five categories we cannot” — with the five listed. Presenting it as a complete picture is worse than having no inventory, because the next question will be answered with unearned confidence.
It is also worth being clear that none of this made us more secure on the ninth of December. We were not exposed, and we did not know that for eleven hours. The inventory buys the knowing, which is a smaller and more honest claim than the one usually made for it.