The DST Root CA X3 expiry and the clients that broke

A root certificate expired on 30 September, and every server with a valid certificate started failing for clients with an old trust store.

$ curl https://example.com
curl: (60) SSL certificate problem: certificate has expired

$ openssl s_client -connect example.com:443 -showcerts
# the chain ends at DST Root CA X3, expired 2021-09-30

# the fix is a chain change, not a certificate change:
# serve the SHORT chain, ending at ISRG Root X1.

The server certificate was valid throughout, which is why the monitoring said everything was fine — expiry checks look at the leaf. Modern clients have ISRG Root X1 in their store and ignore the expired cross-sign; OpenSSL 1.0.2 and older Android do not, and they are exactly the clients on the other end of a server-to-server integration. The lesson worth keeping is that a certificate chain has an expiry independent of the certificate.