Sixty subdomains meant sixty certificates, sixty renewal jobs and sixty opportunities for one of them to fail silently. Two of them had. ACME v2 shipped in March with wildcard support, which collapses all of that into one certificate — and replaces it with a single credential that can rewrite the DNS zone.
The symptom
$ for h in $(cat hosts.txt); do
> printf '%-28s ' "$h"
> echo | openssl s_client -servername "$h" -connect "$h:443" 2>/dev/null
> | openssl x509 -noout -enddate
> done
api.example.com notAfter=Apr 12 09:14:00 2018 GMT
staging.example.com notAfter=Feb 2 11:02:00 2018 GMT ← expired
metrics.example.com notAfter=Jan 18 07:44:00 2018 GMT ← expired
docs.example.com notAfter=May 3 15:20:00 2018 GMTBoth expired hosts had failed the same way: the HTTP challenge could not reach them, because both were behind an authentication proxy that had been added months after the certificate was first issued. Nothing alerted, because the renewal cron reported its failure to root’s mailbox on a machine with no mail transport.
Why it happens
The HTTP-01 challenge proves control of a hostname by serving a file at a well-known path on that hostname. That is a per-name proof, so it can never authorise *.example.com — there is no host to serve the file from.
DNS-01 proves control of the domain instead, by writing a TXT record under _acme-challenge. That is exactly the right proof for a wildcard, and it means whatever performs the renewal needs credentials that can write records in the zone. The certificate is now trivial; the credential is the whole problem.
The fix
Issuing it
$ certbot certonly
--dns-cloudflare
--dns-cloudflare-credentials /etc/letsencrypt/cf.ini
--dns-cloudflare-propagation-seconds 30
--server https://acme-v02.api.letsencrypt.org/directory
-d 'example.com' -d '*.example.com'
Waiting 30 seconds for DNS changes to propagate
Successfully received certificate.
/etc/letsencrypt/live/example.com/fullchain.pemThe explicit --server is needed on older certbot builds because v1 remains the default for a while and v1 cannot issue wildcards; the error when you forget is unhelpful. Listing the apex separately matters too — *.example.com does not cover example.com, and nor does it cover a.b.example.com, since the wildcard matches exactly one label.
The propagation delay is worth setting higher than the plugin’s default. A provider with slow secondaries will occasionally fail validation and succeed on a manual retry, which produces the worst kind of intermittent failure — one that always works when you are watching.
Scoping the credential, which is the actual work
The default API key most DNS providers hand out can do everything to every zone on the account. Putting that on a web server means the web server can repoint the MX record, which is a considerably larger blast radius than the certificate it exists to renew.
# /etc/letsencrypt/cf.ini
dns_cloudflare_api_token = ... # Zone:DNS:Edit, ONE zone, nothing else
$ chmod 600 /etc/letsencrypt/cf.ini
$ chown root:root /etc/letsencrypt/cf.ini
$ ls -l /etc/letsencrypt/cf.ini
-rw------- 1 root root 74 Mar 14 09:12 /etc/letsencrypt/cf.iniScoped tokens exist at every major provider now and there is no reason to use a global key. Certbot warns about the file permissions if they are wrong, which is a good default and is not a substitute for checking.
Warning
The stronger arrangement is a delegated zone: _acme-challenge.example.com is a CNAME to a record in a separate zone that holds nothing else, and the token is scoped to that zone. A compromise of the renewing host then buys the attacker the ability to issue certificates for your domain — which is bad — and not the ability to redirect your mail, which is worse.
Renewal that fails loudly
A renewal timer that has never actually renewed anything will fail in sixty days, and the first sign will be a browser warning. Both halves of that need fixing.
# /etc/systemd/system/certbot-renew.service
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot renew --quiet
--deploy-hook 'systemctl reload nginx'
# /etc/systemd/system/certbot-renew.timer
[Timer]
OnCalendar=*-*-* 03,15:00:00
RandomizedDelaySec=3600
Persistent=true
[Install]
WantedBy=timers.targetThe deploy hook is the half that fails silently in real life: the certificate renews on disk, nginx keeps serving the old one from memory, and nothing is wrong until somebody notices the expiry date in a browser. Twice a day with an hour of jitter is the recommended cadence and the jitter is not decoration — it spreads load across the CA and avoids a thundering herd of renewals at midnight.
$ certbot renew --dry-run
Simulating renewal of an existing certificate for example.com and *.example.com
Congratulations, all simulated renewals succeeded
$ systemctl list-timers certbot-renew
NEXT LEFT UNIT
Mon 2018-03-19 15:41:12 UTC 6h left certbot-renew.timer
$ journalctl -u certbot-renew --since '7 days ago' | grep -c 'Congratulations'
14The dry run uses the staging environment, so it exercises the challenge, the DNS write and the hook without touching the rate limit — which matters, because the production limit of five duplicate certificates a week is easy to hit while debugging. Running the dry run from cron monthly and alerting on a non-zero exit is the check that keeps this honest.
Verifying it worked
$ echo | openssl s_client -servername anything.example.com
-connect anything.example.com:443 2>/dev/null
| openssl x509 -noout -text | grep -A1 'Subject Alternative Name'
DNS:*.example.com, DNS:example.com
$ openssl s_client -connect example.com:443 -showcerts 2>/dev/null
| grep -c 'BEGIN CERTIFICATE'
2 # leaf + intermediate. not 1.
$ ./cert-expiry-check.sh $(cat hosts.txt)
# no output = nothing under 30 daysCounting the certificates in the chain is the check nobody runs and the one that catches a misconfiguration Chrome tolerates and older Android does not — serving only the leaf works on any client that already has the intermediate cached, which is most desktop browsers and not most phones. Using fullchain.pem rather than cert.pem in the nginx configuration is the fix, and the two filenames are similar enough that this happens regularly.
What this costs
One private key now protects every subdomain, so a compromise on any host that has it is a compromise of all of them. Sixty separate certificates had a genuine security property — a stolen key from the staging box did not help against the API — and consolidating trades that away for operational simplicity. On an estate where the hosts have meaningfully different trust levels, the honest answer is a wildcard per trust boundary rather than one for everything.
The second cost is that renewal now depends on the DNS provider’s API, which is one more thing that can be down or can change. A provider deprecating an API version breaks renewals silently and on their schedule rather than yours, and the failure surfaces sixty days later. That is an argument for the dry run in cron rather than against wildcards, but it is a real dependency that did not exist when the proof was a file on a web server.