$RodHat_
Rod's Tales

The certificate renewed and nginx kept the old one.

Published by

Photo by RealToughCandy.com
Photo: RealToughCandy.com / Pexels

The call came in at seven-thirty on a Tuesday morning. HTTPS was broken. Not slow, not intermittent. Broken. Every browser, every client, every curl. NET::ERR_CERT_DATE_INVALID, the universal message that means your certificate expired and you should feel bad about it.

I know what you are thinking. Somebody forgot to renew the cert. That is never the story. The story is always more interesting than that.

Let’s Encrypt had renewed the certificate seventy-two hours earlier, right on schedule, sixty days into the ninety-day lifecycle as designed. Certbot ran, talked to the ACME endpoint, passed the HTTP-01 challenge, got a fresh cert, wrote it to disk. The certbot log said Congratulations, all renewals succeeded. The cert file on disk was valid for another ninety days.

nginx was serving a cert that expired at 07:14 that morning.

These two facts coexisting is how you end up in a situation where your monitoring says everything is fine right up until users start calling.

The diagnosis started with the obvious check. openssl s_client is the tool:

$ openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates
notBefore=Jul 17 12:04:23 2026 GMT
notAfter=Oct 15 07:14:23 2026 GMT

Expired. Yesterday. Okay.

But the cert on disk:

$ openssl x509 -noout -dates \
  -in /etc/letsencrypt/live/example.com/fullchain.pem
notBefore=Oct 13 09:22:11 2026 GMT
notAfter=Jan 11 09:22:11 2027 GMT

Valid until January. So certbot did its job. The file is there. The file is good. And nginx has no idea it exists.

$ nginx -T 2>/dev/null | grep ssl_certificate
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

The config points at the right path. nginx knows the path. It just never reloaded the cert at that path.

This is how TLS in nginx works: the certificate gets read into memory when the process starts or when you send a reload signal. After that, the cert in memory is what gets served, regardless of what is on disk. You can replace the cert file on disk, you can replace it seventeen times, and nginx will keep handing out the old one until you tell it to look again. SIGHUP or systemctl reload nginx or nginx -s reload. One of those. Pick your religion.

Nobody had sent any of those in ninety days.

The certbot post-hook was supposed to handle this. The configuration was in /etc/letsencrypt/renewal/example.com.conf:

[renewalparams]
authenticator = webroot
webroot_path = /var/www/letsencrypt
post_hook = systemctl reload nginx

post_hook runs after every successful renewal. Or it should.

$ systemctl reload nginx
Failed to reload nginx.service: The name org.freedesktop.systemd1 was not provided by any .service files

There it is.

The server had been migrated from Debian 11 to Debian 12 sometime in the spring. The systemd D-Bus socket had a permission change. The certbot renewal timer runs as root. But systemctl reload from a non-interactive root session, after the distro upgrade, needed an explicit --no-pager and an active D-Bus session that was not there. The return code was non-zero. Certbot logged the non-zero exit, noted that the post-hook failed, and marked the renewal as succeeded anyway because the certificate was obtained successfully.

The cert part worked. The reload part failed silently for three renewal cycles.

The part that was supposed to make the renewed cert active never ran.

I want to dwell on the monitoring for a moment, because this is where we could have caught it.

The health check was an HTTPS GET to the root endpoint, checking for HTTP 200. It got HTTP 200. The cert was valid until 07:14 on October 15th, and until that moment, it was technically valid, and nginx was serving it, and the check passed. The alert fired at 07:15 when the first check after expiry failed.

This is the wrong check. The health check that was green for forty-three minutes is the same category: a probe that confirmed process existence instead of measuring the thing that actually expires.

The right check is cert expiry, not HTTPS connectivity. These are different things. HTTPS connectivity tells you the service is up. Cert expiry tells you how long until it goes down on its own without anyone doing anything wrong. The two checks are measuring different failure modes and you need both.

# Cert expiry check: warn at 14 days, critical at 7
EXPIRY=$(echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -enddate 2>/dev/null \
  | cut -d= -f2)
EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
NOW_EPOCH=$(date +%s)
DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
echo "days_until_expiry=$DAYS_LEFT"

With a fourteen-day warning window, the first cert renewal failure would have alerted on day seventy-seven. We had three cycles of failures. Ninety days without a single expiry check firing, because the expiry check did not exist.

If you run a public HTTPS service and your monitoring only checks for a 200 status code, you are going to be here eventually. The cert expiry metric costs nothing to add and it gives you two weeks of lead time on exactly this failure. Add it.

The fix was immediate: systemctl reload nginx. Cert in memory updated to the January cert, browsers happy again, seven minutes of downtime total.

The post-hook fix took a bit more thought. On Debian 12, systemctl from a non-interactive root session under the certbot timer context needs DBUS_SESSION_BUS_ADDRESS to be set, which it is not in a timer unit. The clean solution is to bypass D-Bus entirely:

post_hook = nginx -s reload

nginx -s reload sends SIGHUP directly to the nginx master process by reading the PID file. No D-Bus. No systemd socket. Works in any execution context, works in a chroot, works on FreeBSD where you would use service nginx reload anyway and systemctl does not exist in the first place. If you are already on a system where nginx is managed by systemd, you lose nothing by using the direct signal mechanism in cron and timer contexts. The supervision-without-systemd tip covers the broader approach for services that should not rely on D-Bus being present in the execution context.

Test the post-hook before you need it. Run it manually as root in a non-interactive shell:

$ env -i HOME=/root PATH=/usr/sbin:/usr/bin:/sbin:/bin \
  bash --noprofile --norc -c 'nginx -s reload'

If that command fails, your certbot renewal will “succeed” and your cert will sit on disk while nginx serves the old one. The test takes thirty seconds. The outage costs you however long it takes for someone to call at seven-thirty in the morning.

There is a broader pattern in cert management that trips up systems that were set up correctly and then quietly stopped being correct.

Certbot uses a deploy hook or post-hook to run the reload step. The hook gets tested once, at setup time, in an interactive session where D-Bus and PATH and environment variables are all exactly right. Then the system gets upgraded. Or the service name changes. Or a package manager replaces /etc/letsencrypt/renewal/ with a new version. Or someone moves nginx to a different user. And the hook silently starts failing, certbot silently reports success, and nothing alerts because the cert is still valid for another sixty days, then thirty, then fourteen, then seven, then zero.

The only reliable defense is a monitoring check that measures what you actually care about: how many days until the certificate being served by the live process expires. Not whether HTTPS works. Not whether the cert file on disk is valid. What the process is handing to clients, right now, over the wire.

On the same Tuesday, I checked twelve other services managed by the same team. Four of them had certbot configurations with post_hook = systemctl reload. Three of those four had the same D-Bus problem. None of the twelve had cert expiry monitoring. Two had certs that had already been renewed on disk but not reloaded into the running process. Neither was expired yet, but both were serving ninety-day-old certs while the fresh certs sat unused in /etc/letsencrypt/live/.

The renewal is not the hard part. The reload is the hard part. The reload is the part that runs in a context nobody tested, on a machine that was slightly different from when the hook was written, as a subprocess of a timer unit that has no interactive session and no D-Bus and no good way to tell you it silently failed.

Check what your process is actually serving. Do it with openssl s_client. Do it from outside the box. That is the cert that matters.