The machine that forgot what year it was
Published by RodHat
Kernel upgrades are supposed to be boring. You schedule the maintenance window, tell the on-call rotation to expect a fifteen-minute outage, reboot the box, verify it comes back, close the ticket. I had done this exact procedure on this exact box three times in the past year. The fourth time, the box came back wrong.
Not obviously wrong. It passed every health check. Services came up. The load balancer saw it healthy. Traffic routed to it normally. For about ninety seconds after each reboot, some percentage of requests would fail with TLS errors, and then everything would be fine.
At first I thought it was a race condition in the startup order. Services coming up before the reverse proxy was ready, or the cert chain not being written to disk before the service tried to read it. I added a sleep to the startup script. The ninety-second window stayed exactly ninety seconds. Exact to within a few seconds, which is not what a race condition looks like.
What the TLS errors actually said
I pulled the errors out of the service log instead of reading the alert summary. The summary said “TLS handshake failed,” which is not useful. The actual error was:
x509: certificate signed by unknown authority (tls: certificate verify failed)
x509: certificate has not yet started being valid“Not yet started being valid” is a specific error. It means the certificate’s notBefore field is in the future from the perspective of the machine making the connection. The certificate was issued on September 1, 2026. The machine trying to verify it thought the date was something earlier.
dateWed Sep 6 04:31:18 UTC 2026The system clock looked fine. I ran it immediately after the reboot. Fine. I stared at this for a while.
Then I ran:
hwclock --show2000-01-01 00:00:09.843201+00:00There it was. The hardware clock thought it was January 1, 2000, nine seconds after midnight.
Two clocks
Linux keeps two separate clocks running simultaneously. The system clock is the one everything uses: date, time(), the kernel’s internal timestamp, everything that touches time during normal operation. The hardware clock is the Real-Time Clock (RTC), a battery-backed oscillator on the motherboard that runs continuously even when the machine is powered off. Its job is to preserve the time across shutdowns so that when the machine boots, it has something to read.
At boot, the kernel reads the hardware clock and sets the system clock from it. That is the handoff. After that point, the system clock runs independently, and NTP adjusts it based on external references. The hardware clock sits there in the background, continuing to tick, not consulted again until the next boot (or until you explicitly sync it with hwclock --systohc).
When the CMOS battery dies, the hardware clock loses power when the machine shuts down. When the machine boots, the RTC comes up at whatever the BIOS reset value is: for most hardware, January 1, 2000, 00:00:00. The kernel reads that value and sets the system clock accordingly. The machine starts with its time in the year 2000.
NTP then runs, computes the actual time, and corrects the clock. This takes some number of seconds. During those seconds, the system clock is wrong.
How NTP handles a twenty-six year offset
NTP does not just snap the clock to the correct value every time it checks. Abrupt clock changes break things: timestamps go backward, scheduled jobs misfire, log entries interleave incorrectly, and anything using gettimeofday() to measure elapsed time gets garbage. So NTP normally slews the clock: it adjusts the rate at which the clock ticks, speeding it up or slowing it down, until the offset closes. Maximum slew rate under ntpd is 500 parts per million, which is 0.5 milliseconds per second.
Slewing a 26-year offset at 500ppm would take… I will not make you do the math. It is not a number that makes practical sense. For large offsets at startup, both ntpd and chrony can step the clock: jump it immediately to the correct value, regardless of the magnitude.
Chrony’s config for this is makestep:
makestep 1.0 3This tells chrony: if the offset is greater than 1.0 second, step the clock instead of slewing. Do this for the first 3 clock updates after startup. After the first 3 updates, if the offset is still large, do not step: figure out what is wrong because something is seriously broken.
The “first 3 updates” part is why the correction took ninety seconds. Chrony does not have a measurement immediately at startup. It has to contact the NTP pool, get several responses, build up confidence in the offset estimate, and then apply the step. On this box, with the NTP servers it was configured against, that took about eighty to ninety seconds. During those ninety seconds, the system clock was in the year 2000.
Every TLS connection the service made during those ninety seconds failed because every certificate in our infrastructure had a notBefore date after January 1, 2000. Which is all of them.
What else went wrong in ninety seconds
The TLS failures were the visible symptom. They are not all that went wrong.
Anything that wrote a timestamp during the ninety-second window wrote a timestamp with the year 2000. Log entries from the first ninety seconds of each boot showed up at the beginning of log aggregation time-sorted output because they sorted before everything else. They were not obviously wrong unless you noticed the year.
The cron daemon picked up scheduled jobs based on the current time at startup. With the clock at midnight January 1, 2000, any job configured to run between midnight and 1:30am (the window on a previous boot’s correction) was at risk of running immediately at startup, because the clock said it was within the window when the job was scheduled.
One of those jobs was a database backup rotation script that was supposed to run at 01:00 and delete backup files older than seven days. January 1, 2000 minus seven days is December 25, 1999. Every backup file on the system had a timestamp in 2026. From the script’s perspective, everything was more than seven days old. The script deleted all backup files.
I found this out about six hours later when I tried to verify the backup.
Finding the battery
The dead CMOS battery is not hard to diagnose once you know what you are looking for.
hwclock --showAfter a reboot with a dead battery: 2000-01-01 00:00:XX. After NTP has corrected the system clock, the hardware clock has not been updated. It is still ticking from wherever it started.
hwclock --show
# 2000-01-01 01:34:27.119430+00:00
# (it has been running for 94 minutes since the reset)
date
# Thu Sep 6 04:31:18 UTC 2026The system clock is correct. The hardware clock thinks it is January 1, 2000, 01:34. The divergence between them is 26 years and some change.
On this box, the battery had been dying for a while. The RTC had been drifting forward on every boot, NTP had been fixing it, and nobody had noticed because there was no monitoring on the discrepancy between system clock and hardware clock. The drift had been getting slightly worse each reboot as the battery weakened. Eventually the battery died completely and the clock reset to epoch on shutdown.
The physical replacement took about four minutes. CR2032, $2.50 from the box in the parts drawer. I have had $2.50 hardware failures that cost considerably more than $2.50 to resolve.
Fixing the soft failures
Replacing the battery stops future occurrences. It does not fix the ninety-second window that will persist on any box where there is a large clock offset at startup, for whatever reason.
The right chrony configuration for a physical box that may have RTC issues:
# /etc/chrony.conf
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst
server 3.pool.ntp.org iburst
# Step the clock on startup if offset > 0.1 seconds, for first 3 measurements
makestep 0.1 3
# Sync the hardware clock after correction
rtcsyncrtcsync tells chrony to sync the hardware clock whenever it adjusts the system clock. This keeps the hardware clock calibrated so that a reboot does not start twenty-six years behind.
For ntpd (which does not have makestep), the equivalent is running ntpd -g which allows the initial step regardless of magnitude, and then configuring tinker panic 0 in ntp.conf to disable the sanity limit that would otherwise abort if the offset is too large.
After correcting chrony’s config, the ninety-second window went away on subsequent test reboots. The step happened within eight seconds of startup, before any application services were ready to make outbound connections.
Monitoring what you cannot see from the outside
/proc/driver/rtc or hwclock --show gives you the hardware clock. The gap between it and the system clock is the thing to watch:
# Detect hardware clock drift relative to system clock
hwclock --show --utc | awk '{
cmd = "date -u +%s"
cmd | getline now
close(cmd)
split($1" "$2, a, /[- :]/)
# rough check: if RTC year differs from system year, something is wrong
print "RTC year:", a[1], "System year:", strftime("%Y", now)
}'This is rough, but a monitoring check that alerts when hwclock --show year differs from date year catches a dead battery before the next reboot does it for you. Nagios, check_mk, whatever you run: add it. It checks in under a second and a wrong answer means you have a hardware problem to fix before it becomes a production problem.
For the bigger picture on your TLS debugging toolkit when clock skew is the suspect, the openssl s_client walkthrough covers how to read the notBefore/notAfter fields from a live connection and compare them against what the machine thinks the current time is. Takes thirty seconds and rules out half the TLS failure modes.
The backup
I got the backup back. Most of it.
The rotation script had deleted everything before keeping any new backup for the current run, which is a sequencing bug I will not describe in detail because thinking about it still makes me angry. The S3 bucket had versioning enabled and a thirty-day retention on versions, so I recovered everything the script had deleted. The recovery took about two hours because the CLI does not make bulk version restoration easy.
The deleted-file recovery pattern is about files that are unlinked but still open. This was different: genuinely deleted files recovered from object storage versioning. The principle is the same. You need a path back that does not depend on the same system that lost the files in the first place.
Time as infrastructure
The system clock is not a background detail. It is infrastructure: every certificate validation, every log entry, every scheduled job, every distributed lock, every signed request to any API with a timestamp window depends on it being correct.
Most of the time, NTP handles this automatically and you never think about it. NTP is good. The problem is that NTP is a service, and services have failure modes, and “what happens when NTP cannot correct the clock fast enough” is not a question most monitoring setups can answer.
The box had been rebooting with a wrong clock for long enough that it had become a known weird thing. “The service is slow for a minute after a reboot” was in the team’s muscle memory, attributed vaguely to startup latency, never investigated because it resolved itself. The CMOS battery had been dying for months. Nobody ran hwclock --show after a reboot because nobody thought to, and nothing was checking it.
The battery costs less than a coffee. The investigation cost a day.
hwclock --show after every maintenance reboot. It takes one second and the output is either boring or explains everything that is about to go wrong.