Three certificate incidents in six years, each with a different cause, each followed by a fix that addressed that cause. The third one was an expired certificate on a service nobody remembered existed, which none of the previous two fixes could have prevented because both assumed you know what you are serving.
The symptom
09:41 a supplier reports that a callback endpoint
is failing TLS verification
09:44 the endpoint is not in the deployment
inventory
09:52 it is on a host that is in the inventory, on
port 8443, served by a container nobody
recognised
10:20 identified: an internal API added in 2021 for
one integration, with its own certificate
from a private authority, expired 6 October
the supplier had been failing since the sixth. they
reported it on the fourteenth.Eight days of a failing integration before anybody was told, because the failure was on their side of the connection and their retries were silent. The certificate was not covered by any renewal automation because nothing knew it existed.
Why it happens
Certificate renewal is automated per service, and the inventory of services is maintained by hand. Any service that predates the automation, or that was added without touching it, is outside the system entirely — and there is no signal until it expires.
The fix
The three previous causes
2019 a renewal blocked by a rewrite rule added the
week before → the ACME challenge path is now
excluded, and a check asserts it.
2024 a renewal that succeeded and a certificate that
was never served, because the nginx reload hook
had been failing for three months → an external
check on the SERVED certificate.
2025 a certificate on a service no inventory listed
→ neither previous fix applies, because both
operate on a list of endpoints.Each fix was correct and each addressed the specific mechanism that had just failed, which is the shape of every incident response. The third cause is a category the first two could not reach — they improve the checking of known endpoints and say nothing about completeness.
An inventory built from what is listening
#!/usr/bin/env bash
set -euo pipefail
# what we think we serve
./bin/configured-endpoints | sort -u > /tmp/configured
# what actually answers TLS, across our address space
nmap -Pn -sT --open -p 443,8443,9443 -oG -
"$RANGES" | awk '/Ports:/ {print $2}' | sort -u > /tmp/hosts
for h in $(cat /tmp/hosts); do
for p in 443 8443 9443; do
timeout 5 openssl s_client -connect "$h:$p" </dev/null 2>/dev/null
| grep -q 'BEGIN CERTIFICATE' && echo "$h:$p"
done
done | sort -u > /tmp/actual
comm -13 /tmp/configured /tmp/actualthe first run found two endpoints nobody could account
for:
10.0.1.14:8443 the internal API from 2021, the
subject of this incident
10.0.1.22:443 a staging host from 2021, serving a
copy of the application at the 2021
commit, on PHP 8.1, with a database
containing real customer data copied
for a debugging session in 2022
the second one is a much larger finding than the
certificate.Scanning your own address space feels excessive until it finds a four-year-old staging host serving real customer data on an end-of-life runtime. The certificate incident was the prompt and the staging host was the actual outcome, which is the usual relationship between what an investigation is for and what it produces.
The internal certificate
issued 2022-10-06 by an internal authority
validity three years
expires 2025-10-06
renewed by a documented manual procedure, last
executed by somebody who left in 2023
a three-year validity is a reminder set three years in
advance, and nobody sets those.
the immediate fix was a renewal. the actual fix is a
one-year validity with automation, because a longer
validity is a longer interval between exercises of a
procedure that is guaranteed to have rotted.One check, on everything discovered
for endpoint in $(./bin/tls-inventory); do
host="${endpoint%:*}"
end=$(echo | timeout 5 openssl s_client -connect "$endpoint"
-servername "$host" 2>/dev/null
| openssl x509 -noout -enddate | cut -d= -f2)
days=$(( ( $(date -d "$end" +%s) - $(date +%s) ) / 86400 ))
[ "$days" -lt 21 ] && ./bin/alert "$endpoint expires in $days days"
doneThe check runs from outside and depends on none of the machinery that renews, deploys or reloads — which is the property that matters, because all three previous failures were in one of those. The inventory it iterates is regenerated by the scan rather than maintained, which is what closes the third cause.
The staging host
destroyed the same day, and the follow-up:
the database it held was a copy of production from
2022, with 6,000 customer records, unencrypted, on
a host with no monitoring and an end-of-life PHP.
reachable from the internet on 443.
logged access, from the four years it existed: 41
requests, all of them scanners.
reported internally, and the process change is that a
database copy for debugging now expires — a scheduled
job drops any database on a non-production host older
than 14 days.Forty-one requests over four years, all from scanners, is the best possible outcome from a genuinely bad situation. The expiring-copy job is the process change that matters, and it is the kind that only gets built after somebody has found the thing it prevents.
Verifying it worked
$ ./bin/tls-inventory --diff
configured: 8 discovered: 8 unaccounted: 0
$ ./bin/cert-expiry-check
8 endpoints checked, minimum 47 days
# a deliberately expired test certificate
$ ./bin/cert-expiry-check --endpoint=test.internal:8443
ALERT: test.internal:8443 expires in -2 days
$ ./bin/stale-databases
non-production databases older than 14 days: 0
# and the scan, quarterly since October
runs: 1 unaccounted endpoints found: 2 (both
resolved)Testing the alert with a deliberately expired certificate is the step that distinguishes a check that works from one that has never fired, and it takes ten minutes with a self-signed certificate. Everything else here is a scheduled job whose value is entirely in the first run.
What this costs
A scanner that will find something nobody wants to own. The two endpoints it found in October both had an owner in principle and neither had one in practice, and the next run will find a third — which is a conversation rather than a technical problem and is the reason this kind of check gets quietly disabled.
Scanning your own address space also produces traffic that looks like an attack to any monitoring that is watching, which needed an exclusion. That exclusion is now a hole in the intrusion detection shaped exactly like our scanner, and the honest position is that it is a small hole and it is a hole.