The security advisory hit on a Tuesday. Kernel privilege escalation, local exploit only, proof-of-concept already public. The CVSS was high enough that even the people who usually ignore CVEs were paying attention.
I was not running the fleet in question. I was a consultant, brought in afterward to explain what happened. What happened was straightforward, in that way that makes you want to scream.
The operations team was competent. They had a patching process. When the advisory came in, they opened a maintenance ticket, ran the playbook: apt-get update && apt-get upgrade -y across the fleet, verified package versions, closed the ticket. Done inside the maintenance window. Change review signed off. The compliance checkbox got checked.
Two weeks later, a contractor doing a scheduled pentest reported that they had root on four of the sixteen boxes.
The operations team was confused. The kernel had been patched. They had the ticket. They had the package versions. They ran dpkg -l 'linux-image-*' again, right there in the incident channel, and pasted the output. Current. All of them. The patched version, exactly as advertised.
Then someone ran uname -r.
$ uname -r
5.15.0-91-genericThe patched package installed version 5.15.0-105-generic. The boxes had been running 5.15.0-91-generic for two months straight without a reboot. The package was current. The running kernel was not. Nobody had rebooted.
This is where I came in, because they could not explain the discrepancy and wanted a third party to either confirm it or find something worse.
There was nothing worse. This was the whole thing.
Here is how kernel updates work on Linux, for the people in the back:
When you run apt upgrade and a new kernel package is downloaded, it gets unpacked into /boot. The new vmlinuz image sits there, the new initrd gets generated, the bootloader config gets updated. All of that happens on disk. None of it touches the running kernel. The running kernel is loaded into memory at boot and stays there until the next boot. dpkg -l tells you what is installed on disk. uname -r tells you what is running in memory. These are different questions and they can have different answers.
This is not a quirk. This is not a bug. This is how Unix has worked since before most of the people on that operations team were born. The kernel is not a daemon you can reload with a signal. It is not a config file you can SIGHUP. You replace it by rebooting into the new one.
There is no other way, unless you are using kexec, which bypasses the bootloader and jumps directly to a new kernel image. kexec is real, it is in production at places that cannot tolerate the full POST and BIOS cycle, and it still requires the running process to explicitly hand off. It is not what apt upgrade does.
On RPM-based systems, needs-restarting -r will tell you if a reboot is required after updates. It checks the kernel package timestamp against the running kernel’s boot time. If you patched a kernel, it knows:
$ needs-restarting -r
Core libraries or services have been updated since boot-up:
* kernel
Reboot is required to fully utilize these updates.On Debian and Ubuntu, the equivalent is /run/reboot-required. If that file exists after an upgrade, the system is telling you it needs a reboot. It does not force one. It does not alert anyone. It writes a file and waits for you to notice. The file does not email the operations team. The ticket does not check for it. The playbook, in this case, did not mention it.
You can also check it directly:
# installed kernel
dpkg -l 'linux-image-*' | grep '^ii' | sort -V | tail -1
# running kernel
uname -r
# if these do not match: reboot requiredOne line. Two commands. Neither was in the playbook.
The thing that made this sting, beyond the obvious, was that the team had done something right that most teams don’t: they patched quickly. The CVE dropped, they moved within the maintenance window, they verified the packages. The failure was in the verification step. Checking the package version is checking what got installed on disk. The threat model is about what is running in memory.
There is a similar story with certificates: the new cert gets written to disk, but the running daemon still holds the old one in memory because nobody sent it a signal. I wrote about that one a few weeks ago. The pattern is the same. The disk and the process have different views of the world, and the process wins every time because the process is the thing actually doing the work.
The complementary failure is sysctl tuning that doesn’t survive a reboot: your changes run fine until the box reboots, and then you discover the kernel parameters you tuned were never written to /etc/sysctl.d/. Same gap between disk and runtime state, opposite direction.
The recommendation I put in the post-incident report was simple:
- After any kernel package upgrade, verify with
uname -rthat the running kernel matches the installed package before closing the ticket. - On Debian/Ubuntu systems, check for
/run/reboot-requiredas part of the patching playbook. - Schedule the reboot. Do not assume the upgrade completed the patch.
The operations manager asked if there was a tool that could enforce this automatically, centrally, across the fleet.
There are several. Canonical sells Livepatch. RHEL has kpatch. Both apply kernel patches without rebooting by using runtime kernel patching techniques. Both cost money and have limitations on what they can patch. Both are better than a patching playbook that treats dpkg -l as the final word on what is running.
But the three-line shell check above is free, always works, and takes six seconds. If your playbook has room for twenty compliance checkboxes, it has room for uname -r.
The four boxes that got popped were development infrastructure. Nothing critical. The contractor was authorized. The damage was a pentest report and a bad afternoon, not a breach.
The next time you run a kernel upgrade, before you close the ticket: uname -r. One command. Thirty characters. Compare it to the package you just installed.
If they don’t match, you’re not done. You just staged the fix. You haven’t applied it yet.