Certificates that renew themselves

Let’s Encrypt left beta in April, which changes the economics of TLS rather than the technology. A certificate is now free, issued in seconds by an API, and valid for ninety days — and that last number is the interesting one, because ninety days is short enough that renewal has to be automated whether or not anyone wanted to automate it.

The symptom

A staging environment served an expired certificate for six days before anyone mentioned it. The certificate had been bought, installed, and forgotten by someone who had since left; the calendar reminder was in their calendar.

$ echo | openssl s_client -connect staging.example.com:443 2>/dev/null 
>   | openssl x509 -noout -dates
notBefore=Mar 14 00:00:00 2015 GMT
notAfter=Mar 14 23:59:59 2016 GMT

Why it happens

Annual manual renewal is a task with no owner and a twelve-month feedback loop. Nothing about the system reminds anyone, the person who did it last time may not be there, and the failure is invisible until it is total. Shortening the validity period to ninety days does not make this worse — it makes it impossible to ignore, which forces the automation that should have existed anyway.

The fix

Issuance, once

The webroot challenge writes a file into the document root and asks the authority to fetch it over HTTP, which proves control of the hostname without stopping the web server.

$ sudo certbot certonly --webroot -w /var/www/shop/public 
>   -d shop.example.com -d www.shop.example.com 
>   --agree-tos -m [email protected] --non-interactive

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/shop.example.com/fullchain.pem

# the challenge path must be reachable before the certificate exists
location ^~ /.well-known/acme-challenge/ {
    root /var/www/shop/public;
    default_type "text/plain";
}

server {
    listen 443 ssl http2;
    ssl_certificate     /etc/letsencrypt/live/shop.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/shop.example.com/privkey.pem;
}

The ^~ matters: without it a regex location for static files can match the challenge path first and serve a 404, which produces a validation failure that says nothing about nginx.

Renewal, forever

$ sudo cat /etc/cron.d/certbot
17 3,15 * * * root certbot renew --quiet --deploy-hook "systemctl reload nginx"

$ sudo certbot renew --dry-run
Congratulations, all renewals attempted to succeed

Twice a day because renew does nothing for a certificate with more than thirty days left, so running it often costs nothing and gives twenty-nine days of retries before anything is urgent. At seventeen minutes past rather than on the hour, because a large number of servers renewing on the hour is a load pattern the service does not need.

--deploy-hook runs only when something was renewed. --post-hook runs every time, which reloads nginx twice a day for no reason and makes the log useless for telling whether a renewal happened.

Verifying it worked

A dry run proves the mechanism. It does not prove the mechanism will still work in eighty-nine days, which is the actual question — so the certificate needs monitoring that does not depend on the thing being monitored.

$ cat /usr/local/bin/check-cert
#!/bin/sh
days=$(( ( $(date -d "$(echo | openssl s_client -connect $1:443 2>/dev/null 
  | openssl x509 -noout -enddate | cut -d= -f2)" +%s) - $(date +%s) ) / 86400 ))

[ "$days" -lt 21 ] && echo "CRITICAL: $1 expires in $days days" && exit 2
exit 0

Twenty-one days is the useful threshold: renewal starts at thirty, so an alert at twenty-one means two weeks of failed attempts have already happened and there is still a week to fix it by hand. Checking from outside, over the network, is the point — a check that reads the file on disk will pass while nginx serves something else entirely.

The configuration certbot does not write

A certificate makes the connection encrypted. It does not make the configuration good, and the nginx defaults of this era accept protocols and ciphers that any scan will mark you down for.

ssl_protocols             TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers               'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:!aNULL:!MD5';

ssl_session_cache    shared:SSL:10m;
ssl_session_timeout  1h;
ssl_session_tickets  off;

ssl_dhparam /etc/nginx/dhparam.pem;

The session cache is the line with a measurable effect on ordinary traffic: without it every returning visitor pays for a full handshake, and ten megabytes holds around forty thousand sessions. Turning session tickets off is the conservative choice — they are faster, and their keys are not rotated by default, which quietly undermines forward secrecy.

Generating the Diffie-Hellman parameters takes a few minutes and happens once. Skipping it leaves nginx on a 1024-bit default, which is the single most common finding on a configuration that is otherwise correct.

$ openssl dhparam -out /etc/nginx/dhparam.pem 2048
$ nginx -t && systemctl reload nginx

$ curl -s 'https://api.ssllabs.com/api/v3/analyze?host=shop.example.com' 
>   | python -c 'import sys,json; print(json.load(sys.stdin)["endpoints"][0]["grade"])'
A

The chain, which is the other half of the mistake

The most common Let’s Encrypt misconfiguration is not the renewal. It is serving the leaf certificate without the intermediate — which every desktop browser papers over by fetching it separately, and many mobile clients and command-line tools do not.

# wrong: the leaf only. fine in Chrome, fails in curl and on older Android
ssl_certificate /etc/letsencrypt/live/shop.example.com/cert.pem;

# right: leaf plus intermediate, which is what fullchain.pem is
ssl_certificate /etc/letsencrypt/live/shop.example.com/fullchain.pem;

certbot writes four files and the names do not help. cert.pem is the leaf, chain.pem is the intermediate, fullchain.pem is both, and privkey.pem is the key. nginx wants fullchain.pem; the ssl_trusted_certificate directive used for OCSP stapling wants chain.pem. Swapping those two is the other half of the mistake, and it produces stapling that silently does nothing.

# verify from outside, with a client that does not fetch the chain for you
$ openssl s_client -connect shop.example.com:443 
>   -servername shop.example.com < /dev/null 2>&1 | grep -E 'depth=|Verify return'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
depth=0 CN = shop.example.com
Verify return code: 0 (ok)

Three levels and a return code of zero is what correct looks like. A chain stopping at depth 0 with an unable-to-verify code is the leaf-only mistake — and it will have been working perfectly in every browser anybody tested with, which is why it survives to production.

What this costs

There is now a dependency on a third party in the path to serving traffic, on a ninety-day cycle. That is a real trade against a purchased certificate valid for a year, and it is worth being clear-eyed about: an outage at the authority during your renewal window is someone else’s incident happening to you.

The mitigation is the monitoring above rather than a fallback, because there is no fallback — twenty-nine days of retry attempts is a wide enough window that any plausible outage fits inside it. What is not acceptable is discovering the automation broke on the day it expires, and that is a monitoring failure rather than a certificate one.