The storage was fine. We were swapping.

The storage vendor’s support ticket is still technically open. I never closed it. The least I can do is let it die quietly on its own.

The setup

The host was a read-heavy cache layer. Large ZFS dataset, mostly immutable objects that got read frequently and written once. The workload was a good fit for ZFS: high ARC hit rates, fast sequential reads for misses, predictable enough that we’d never felt the need to tune much. It had been running clean for about two years.

Then the dataset grew. Not alarmingly — a few hundred gigabytes over about six months, the kind of growth you don’t notice because it’s a few percent a week. The host had 64GB of RAM. The dataset was approaching 80GB of objects that were actually hot. The ARC was doing what ZFS ARC always does: using every byte of RAM it could get, caching aggressively, giving memory back slowly when asked. This is the correct behaviour. This is what you want.

Nobody had noticed that “giving memory back slowly when asked” now had competition it hadn’t had before.

What we saw

Read latency started climbing on weekday afternoons. Not error-rate climbing — the service was returning data, requests were completing — just slower. P99s that had been stable at 8ms started going to 12ms, then 20ms, then on bad afternoons to 60ms or more. The shape was suspicious: it got worse under load and recovered overnight, which pointed at something resource-constrained and self-healing.

The obvious thing to check was storage. We checked storage.

zpool status reported clean. No errors, no degraded vdevs, nothing. smartctl on the drives came back healthy. iostat -x 1 showed elevated await times — average read latency at the block device level was definitely higher than baseline. So the problem was in the storage layer somewhere. Probably.

We disabled scrubs in case they were contending. No change. We looked at the ZFS I/O scheduler, which on Linux runs through the block layer. Tried changing the scheduler from none to mq-deadline. No measurable difference. Checked the ZFS module version — there had been a minor update recently — rolled it back in a test window. Nothing.

By week two we had opened the support ticket. The storage vendor sent a field engineer with their own diagnostics tooling. He ran tests for an afternoon. The drives were fine. The controller was fine. The firmware was current. He filed his own report confirming it and left.

The pivot

Somewhere in week three I was watching the host under load with vmstat 1 — no particular hypothesis at that point, just staring at the numbers waiting for something to be wrong in a way I hadn’t seen yet. The vmstat output on Linux puts swap columns in the middle: si and so, swap in and swap out, pages per second.

I’ve looked at vmstat output probably ten thousand times. When you’re not expecting swap activity, those columns are background. They’re usually zero. They blend into the rest of the line.

They weren’t zero.

Not dramatically not-zero. Not numbers that would make you sit up if you were expecting them. But consistent — every second, a few hundred kilobytes in, a few hundred kilobytes out, ticking along quietly under every read-latency spike.

swapon --show. One entry: /swapfile, 16GB, type file, in use.

ls -la /swapfile. Modification date: seven months ago.

At that point it took about thirty seconds to pull up the infrastructure repository and search the commit log. One commit, seven months prior, message: add swapfile as safety net after mem-pressure incident on cache-host-01 [INFRA-2847]. The ticket was closed. The person who opened it no longer worked there.

What was actually happening

The mechanism, once you see it, is not subtle.

ZFS ARC doesn’t allocate memory the way a normal application does. It integrates with the kernel’s memory management to use the page cache — it fills available RAM with cached data, and when the kernel needs memory for something else, ARC shrinks. The key word is gradually. ZFS ARC release is not instantaneous.

With 64GB of RAM and a hot working set approaching 80GB, the ARC was under real pressure. It was caching as much as it could, but it couldn’t cache everything, so miss rates were slowly climbing. Under that pressure, the kernel’s memory manager — on this host, with vm.swappiness set to 30 (somebody had tuned it down from the default 60, but hadn’t zeroed it) — decided that swapping some anonymous memory was a reasonable way to make space for ARC.

The anonymous memory it chose to swap was the application’s own heap. The application making the read requests to the ZFS pool. Its internal data structures, connection state, request queues, response buffers.

So when a read request came in, the application reached for memory that had been paged out, triggered a swap read from the swapfile (which was on the same disk pool as the data), waited for that, then issued the ZFS read, which was also going to the same pool. Two I/O operations where there should have been one lookup. The storage metrics showed combined I/O that looked like elevated read latency, because it was, technically — just not the read latency anyone was measuring.

The ZFS pool was not slow. We’d been accurately measuring the wrong thing for three weeks.

We should have caught it earlier with bpftrace on the page fault path. The faults column in vmstat was elevated the whole time — page faults on application memory — but we weren’t treating that as a storage signal, so we didn’t look.

The fix

swapoff /swapfile

Read latency dropped within two minutes. The p99 that had been climbing for three weeks fell back below 10ms before the shell prompt returned. I’ve had few more satisfying moments at a terminal.

We removed the swapfile, pulled the entry from /etc/fstab, deleted the file, and set vm.swappiness=0. We also capped zfs_arc_max explicitly at 56GB — leaving 8GB for the application and OS — which we should have done when the dataset started growing, and which would have forced the ARC to evict cold entries rather than relying on kernel pressure.

The support ticket remained open.

The things worth remembering

vm.swappiness=0 does not disable swap. This is a surprisingly common misconception. It biases the kernel heavily toward reclaiming page cache before swapping anonymous memory, but under sufficient memory pressure it will still swap. If you want swap off, run swapoff. If you don’t want swap to be available at all, don’t have a swapfile.

If your host has swap and you didn’t consciously add it, it’s a time bomb. Any workload that puts real pressure on RAM will eventually activate it. ZFS ARC hosts are particularly susceptible because the ARC’s gradual release behaviour means the kernel reaches for swap before ARC has fully shrunk, and both I/O streams hit the same storage.

Any host configuration change that isn’t in version control is invisible. We found the swapfile in git because whoever added it had the discipline to commit it. If they’d just run mkswap and swapon at the shell and walked away, we would have had no breadcrumb at all. The commit that identified the problem was written by a person who no longer worked at the company and hadn’t thought about it since. Version control is for the next poor bastard, not for you.

Check vmstat and actually look at all of it. The si/so columns are easy to skip. They’re usually zero. They’re small. The string of numbers in the middle of the output is dense enough that eyes slide past columns that aren’t alarming. I’d looked at that host’s vmstat output at least a dozen times during the investigation and had not consciously registered swap activity until I was sitting there with nothing else to look at.

The metric that would have ended this in day two was boring, standard, available without installing anything, and printing to a terminal every second for three weeks while we opened support tickets with the storage vendor.

The storage was fine the whole time.

---

The swapfile has since been deleted from every other ZFS cache host in that infrastructure. Each one had swap disabled as part of the initial build spec — except, for reasons that the git log does not explain, cache-host-01, the oldest of them, which predated the spec.