The OOM killer was doing its job

Five weeks. That’s how long the service had been leaking before anyone noticed. Not because we had no monitoring — we had plenty of monitoring. Not because the memory graphs were missing — they were right there, in the dashboards, being recorded faithfully. The service was leaking, the kernel was responding to the leak, the monitoring gaps were in the data the whole time. We just never looked at the right thing.

The OOM killer was doing exactly what it was supposed to do. That’s the whole story.

The pattern I wasn’t looking at

It started — the way these things always start — with a routine check I wasn’t really doing out of suspicion. I was in the metrics dashboard for something else, tabbing through panels, and I noticed the RSS graph for one of our processing nodes looked like a staircase. Not a smooth line. A flat line that stepped up each morning and then held flat through the day.

Twenty megabytes. Thirty. Fifty by the end of the week.

The service was leaking. Slowly, but consistently, and consistently is worse than quickly because quickly causes an incident and incidents get fixed. Slow leaks just accumulate.

I set an alert: page me when heap utilization crosses 80%. I made a note in my task list to look at the service code on Monday. I did not look on Monday, because something else was on fire on Monday, but the alert would tell me when it mattered.

The alert did not fire.

Three weeks later I was back in the same dashboard for the same unrelated reason and the staircase was still there. Still climbing. Eighty megabytes above where it had started. My 80% alert should have fired weeks ago. I checked the alert config. It looked correct. I checked whether alerts were working at all. They were — two other alerts had fired that week. I checked the specific rule. The rule was fine.

I checked the heap utilization metric.

It was flatlined. Complete gap in the data every night, from roughly 2:40am to 2:41am, then resumes. Every single night for at least five weeks, based on how far back the data went.

Thirty seconds of missing data, then the metric comes back, then the heap utilization is lower than where it left off.

I stared at this for a while before I understood what I was looking at.

The thing the kernel was doing

The monitoring agent was dying every night at 2:40am. Not from a bug. Not from a crash. From the OOM killer.

The sequence:

The processing service leaks memory through the business day and into the evening. By 2:30am, when the nightly batch job kicks off on the same node, memory pressure climbs sharply. The kernel starts reclaiming pages. At some point, reclamation isn’t enough, and the kernel’s out-of-memory killer wakes up, runs its scoring heuristic, and selects a victim.

The service: 1.8GB RSS, oom_score_adj set to -500 by the service’s systemd unit, because someone had done the right thing and tuned the unit to protect the service from OOM killing. The monitoring agent: 180MB RSS, default oom_score_adj of 0.

The OOM killer chose the monitoring agent. Every single night.

The monitoring agent’s systemd unit had Restart=always, so it came back up in about thirty seconds. The metric resumed. The heap utilization metric, which the agent was reporting, reflected the agent’s own heap — not the leaking service — so it came back at a low value after the restart, never approaching the 80% threshold that would have paged me.

The alert was correct. The thing it was alerting on was the wrong thing to alert on. The OOM killer was making the rational choice given its inputs. The leak continued.

What the OOM scorer actually does

Here is how the Linux OOM killer scores a process, in brief, because it is worth knowing:

score = (pages_used / total_physical_pages) * 1000
score += oom_score_adj

The process with the highest score gets killed. oom_score_adj runs from -1000 (never kill) to +1000 (kill first), with 0 as default. Child processes inherit the parent’s adjustment.

The service’s -500 adjustment cut its score roughly in half. The 1.8GB service, on a 4GB node, was using 45% of physical memory — a raw score of around 450. Minus 500: negative score. The kernel will not kill a process with a negative OOM score when there are positive-score alternatives available.

The monitoring agent’s 180MB, on the same node: raw score of around 45. Adjustment of 0: score of 45. Not the most attractive target — there were other processes with higher scores — but on nights when everything else had been killed first, or when the scoring worked out a particular way, the monitoring agent lost.

You can read any process’s current OOM score:

cat /proc/$(pgrep monitoring-agent)/oom_score

I ran this at 2:35am on the night I finally understood what was happening. It read 47. I ran it again at 2:41am, after the agent restarted. The process it reported was a new PID.

The things we should have had

Three things were missing, and each absence hid the others.

We didn’t monitor the monitor. The monitoring agent had no heartbeat check. Nothing verified that it was running and reporting. When it died and restarted, the thirty-second gap was invisible unless you specifically looked for it. When observability infrastructure fails silently, you lose the signals that would tell you it failed. A monitoring agent is observability infrastructure. Its aliveness is a thing worth monitoring, separately, on a different path, not through itself.

The OOM adjustment was set without a corresponding commitment to watch the agent. Protecting the service from OOM killing is correct. The person who set oom_score_adj=-500 in the unit file made a reasonable call. But the implication of that call is that something else will be killed first when memory pressure peaks, and that something needs to be accounted for. We didn’t account for it. We assumed the node had enough headroom and moved on.

The memory alert was watching the wrong process. I had set an alert on the monitoring agent’s heap utilization because that was the metric the agent exposed about itself. What I wanted was an alert on the service’s resident set size, collected independently, so that even if the agent died I had a last-known value and a gap I could alert on. The metric that matters needs to come from a process that doesn’t have a stake in the outcome of the event it’s measuring.

The fixes

We set oom_score_adj=500 in the monitoring agent’s systemd unit. Not -1000 — we’re not pretending the agent should never die. If the system genuinely runs out of memory and something has to go, fine. But we want the leaking service to be chosen before the agent that’s reporting on it. Flipping the agent to +500 means it’s a high-value OOM target, which sounds backwards until you think about what you want to happen when memory pressure peaks: kill the least-critical thing, and “least critical” here is the agent, not the service.

We added a heartbeat check: a separate lightweight process pings the monitoring agent’s health endpoint every sixty seconds and writes the result to a different time-series path. If the heartbeat misses three consecutive checks, it fires an alert through a separate channel — not through the monitoring agent, which might be dead. The heartbeat process is 8MB, permanently protected with oom_score_adj=-900. It does one thing and it’s allowed to be precious about it.

We moved the service RSS metric collection out of the agent and into node-level accounting — a separate scrape that reads /proc/<pid>/status directly and reports it to the time-series store under a different key. The scrape is stateless and runs as a cron, so even if it misses one cycle due to an OOM event, the metric path is independent of the monitoring agent’s health. An alert on this metric fires if RSS exceeds a threshold or if the metric has a gap longer than two minutes.

We fixed the leak. That part took a week of reading the process’s actual memory maps — pmap -x, smaps, the usual archaeology — and it turned out to be a third-party library allocating buffers it didn’t release across certain error paths. The fix was boring: catch the error, free the buffer, done. Classic.

The post-mortem math

Five weeks. If the leak was averaging 4MB per night — conservative estimate based on the staircase slope — the service had accumulated about 140MB of leaked memory by the time we found it. On a 4GB node, that’s not catastrophic. But the batch job that ran at 2:30am was adding significant transient pressure, and the combination was reliably pushing the node into OOM territory. On nights when the batch workload was heavier, I suspect things were worse. I don’t have complete visibility into those nights because the monitoring agent was dead.

The thing that actually concerns me, in retrospect, is how well-hidden this was. We had a monitoring system. The monitoring system was recording data. The alerts were correctly configured against the data they had. The OOM killer was behaving rationally. Every individual component was doing what it was supposed to do.

The failure was structural: we had tuned the OOM scoring to protect the service without considering what would be killed instead, and what we’d chosen to kill was the thing watching the service. The monitoring system was watching the wrong process, so when the right process started failing, the watch was blind. The OOM kills happened on a schedule that was too short to page anyone and too long to be conspicuous.

Five weeks of nightly incidents, every one of which lasted thirty seconds, every one of which was cleaned up automatically, every one of which was invisible in aggregate because we had no view that showed us thirty-second gaps in monitoring coverage.

The kernel’s OOM killer knows what it’s doing. It’s been doing it since 2.6. It does not know what you’re doing — it has a heuristic and a score and it picks the highest-score victim, full stop. If you tune the score on a service without thinking about what that score implies for everything running next to it, the kernel will make a locally rational choice that is globally destructive, and it will do so consistently, reliably, and without comment.

oom_score_adj is a knob. Know which direction you’re turning it, and know what you’re turning it toward.