At 15:01 on the thirtieth of September three integration partners stopped being able to reach our API. Our certificate had been renewed eleven days earlier, was valid for another seventy-nine days, and worked perfectly in every browser anybody tried. The monitoring said the endpoint was healthy because the monitoring was a modern client.
The symptom
# from a partner's server
$ curl https://api.example/v1/orders
curl: (60) SSL certificate problem: certificate has expired
# from a laptop, at the same moment
$ curl -sI https://api.example/v1/orders | head -1
HTTP/2 200
$ openssl x509 -in cert.pem -noout -dates
notBefore=Sep 19 00:00:00 2021 GMT
notAfter=Dec 18 23:59:59 2021 GMT
# the leaf is fine. it was never the leaf.Every expiry check in the monitoring looked at the leaf certificate, which is what everybody means by “the certificate” and is one of three or four in a chain. The thing that expired was a cross-signed root from 2000, which we did not issue, do not control and had never thought about.
Why it happens
A TLS chain is a leaf, one or more intermediates, and a root that the client already trusts. A newer certificate authority is trusted by newer clients directly and, for older ones, presents a chain that terminates at an older root the client has had for years.
That older root expires on its own schedule, independent of your certificate. A modern client sees the newer root in its own store, ignores the expired cross-sign and succeeds. An older one has only the expired root and fails.
The fix
Seeing the chain the server actually sends
$ openssl s_client -connect api.example:443 -servername api.example
</dev/null 2>/dev/null | grep -E '^ *[0-9] s:|^ *[0-9] i:'
0 s:CN=api.example
i:C=US, O=Let's Encrypt, CN=R3
1 s:C=US, O=Let's Encrypt, CN=R3
i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
2 s:C=US, O=Internet Security Research Group, CN=ISRG Root X1
i:O=Digital Signature Trust Co., CN=DST Root CA X3 ← expired
$ openssl x509 -in dst-root.pem -noout -enddate
notAfter=Sep 30 14:01:15 2021 GMTThe chain the server sends is not necessarily the chain in the configuration file — a reload may have been missed, a load balancer may terminate TLS with its own copy, or the ACME client may have written a different variant than expected. Reading it off the wire is the only reliable check.
Certificate 2 in that output is the piece that mattered: the ISRG root, cross-signed by DST Root CA X3 so that clients without the newer root would still trust it. On the thirtieth of September that cross-sign stopped being valid.
Serving the short chain
# the long chain: leaf, R3, ISRG X1 cross-signed by DST X3
# the short chain: leaf, R3 only — the client supplies
# ISRG X1 from its own trust store
ssl_certificate /etc/letsencrypt/live/api/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/api/privkey.pem;
# certbot --preferred-chain "ISRG Root X1"
# → fullchain.pem omits the expired cross-sign
# the trade, stated plainly:
# long chain → old clients WITH DST X3 fail after 30 Sep
# short chain → old clients WITHOUT ISRG X1 fail alwaysThere is no chain that satisfies everybody, which is the uncomfortable part. Switching to the short chain fixed the two partners running OpenSSL 1.1.1 and would have broken anybody still on OpenSSL 1.0.2 without an updated trust store — so the decision needed to know which clients existed.
# and finding out, from the access log, which is the only
# evidence available
$ awk -F't' '{print $NF}' /var/log/nginx/api-ssl.log
| sort | uniq -c | sort -rn
88104 TLSv1.3 TLS_AES_256_GCM_SHA384
4102 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
41 TLSv1.2 ECDHE-RSA-AES128-SHA256 ← the old ones
# log_format ssl '... $ssl_protocol $ssl_cipher';Logging the protocol and cipher had been added in 2019 for an unrelated audit and turned out to be the only record of who was connecting with what. Without it the decision would have been a guess, and the guess would have been wrong in the direction of assuming everybody was modern.
Reproducing it, which needs an old client
# a laptop cannot reproduce this. a container can.
for img in centos:7 ubuntu:18.04 debian:9 alpine:3.9; do
printf '%-14s ' "$img"
docker run --rm "$img" sh -c
'curl -sS -o /dev/null https://api.example/health 2>&1
&& echo ok' 2>&1 | tail -1
done
# centos:7 curl: (60) certificate has expired
# ubuntu:18.04 ok
# debian:9 curl: (60) certificate has expired
# alpine:3.9 okFour images and eleven lines is the entire reproduction, and it should have existed before the incident rather than during it. Keeping it as a scheduled job means a compatibility break is discovered by a machine rather than by a partner.
Monitoring the whole chain
# the blackbox exporter reports the EARLIEST expiry in
# the chain, which is the number that matters
- alert: CertificateChainExpiringSoon
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 21
for: 1h
labels: { severity: ticket }
annotations:
summary: '{{ $labels.instance }} chain expires in
{{ $value | humanizeDuration }}'
- alert: CertificateChainExpiringUrgently
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 5
labels: { severity: page }Measuring the earliest expiry across the chain rather than the leaf is the change, and it is one word in a metric name. With the leaf-only check the dashboard showed seventy-nine comfortable days right up to the moment three partners stopped working.
It would still not have alerted on this particular expiry, because the cross-sign was not in the chain we served until certbot chose to include it — which is an argument for the compatibility container test as well as the metric. The two together cover it; either alone does not.
Verifying it worked
$ openssl s_client -connect api.example:443 </dev/null 2>/dev/null
| grep -c 'DST Root CA X3'
0
$ ./bin/tls-compat-check
centos:7 ok
debian:9 ok
ubuntu:18.04 ok
alpine:3.9 ok
$ curl -sG http://prometheus:9090/api/v1/query
--data-urlencode 'query=(probe_ssl_earliest_cert_expiry - time()) / 86400'
| jq -r '.data.result[0].value[1]'
79.2The compatibility check passing on all four is the outcome, and the two partners were back within forty minutes of the chain change. The third took two days because their own trust store was from 2016 and needed updating on their side, which is a conversation rather than a fix.
What this costs
Knowing your clients’ trust stores, which is a thing nobody wants to know and which cannot be avoided if you run a server-to-server API. The cipher log is the cheapest available proxy and it tells you about protocol versions rather than about root certificates — so it narrows the question without answering it.
The container compatibility suite is four images that will themselves become obsolete, and keeping it current means periodically asking which old platforms still matter. That is a maintenance task with no visible benefit until the day it has one, which is the category of work that quietly stops happening.