The host had sixteen gigabytes free and the container had none

04:06. The API service died. Not degraded: dead. Three restarts in twelve minutes, each time climbing through startup and then crashing before it could serve traffic. The monitoring dashboard showed memory usage climbing steadily in the thirty seconds between restart and crash, never leveling off.

The host had 16 gigabytes of RAM. free -h showed 11 gigabytes available. The service’s own metrics reported less than 800 megabytes in use before the crash. Host memory pressure: zero. Something was killing a process that had room to breathe.

What killed it

dmesg -T | grep -i "oom\|cgroup" | tail -20

[Mon Sep 21 04:05:52 2026] Memory cgroup out of memory: Killed process 47291 (api-service) total-vm:1024132kB, anon-rss:511624kB, file-rss:23144kB
[Mon Sep 21 04:05:52 2026] oom_kill_process: Kill process 47291 (api-service) score 1000 or sacrifice child

Memory cgroup out of memory. Not system OOM. Cgroup OOM. The process was inside a memory cgroup with a hard limit. The limit was hit. The kernel killed the process. The host’s 11 free gigabytes were irrelevant.

Host OOM and cgroup OOM are different events, tracked separately, reported in the same place. dmesg | grep -i oom catches both. The phrase “Memory cgroup out of memory” is cgroup OOM. “Out of memory: Killed process” without the cgroup prefix is system OOM. One means the host ran dry. The other means a container hit its ceiling. Both show up in dmesg. Only one of them correlates with host memory metrics.

Where the limit was

docker inspect api-service | jq '.[0].HostConfig.Memory'

536870912

536,870,912 bytes. 512 megabytes.

docker stats api-service --no-stream --format "{{.MemUsage}}"

511MiB / 512MiB

The container was configured with a 512-megabyte memory limit. The service needed 800 megabytes. These two facts were not compatible.

grep -n "mem" docker-compose.yml

    mem_limit: 512m

git log --all -p -- docker-compose.yml | grep -A3 -B3 "mem_limit" | head -20

commit a8f3d119
Author: Jenkins Auto-Deploy <ci@company.internal>
Date:   Thu Oct 17 2024 14:22:08

+    mem_limit: 512m

October 2024. Eleven months ago. The compose file was generated by an internal provisioning tool that pulled a standard template, and the template had mem_limit: 512m because some engineer had set that for a reference service that was small and stateless. Nobody reviewed the limit. The value landed in every service that used the template. Most services ran fine under 512 megabytes. This one eventually did not.

For eleven months the service had stayed under 512 megabytes. Three releases ago, a new feature loaded a reference dataset into memory at startup. The dataset was 340 megabytes. Baseline memory went from 320 megabytes to 660. Still under the limit. Then caching got more aggressive under load: 800 megabytes at normal throughput, 900 at peak. The limit had been wrong for eleven months. It had just taken eleven months for that wrongness to matter.

Reading the cgroup directly

The memory limit is not a Docker artifact. Docker is a wrapper around cgroups. The limit lives in /sys/fs/cgroup/. You can read it without Docker installed, without any agent, without anything except shell access.

CID=$(docker inspect --format '{{.Id}}' api-service)

cat /sys/fs/cgroup/memory/docker/${CID}/memory.limit_in_bytes
# 536870912

cat /sys/fs/cgroup/memory/docker/${CID}/memory.usage_in_bytes
# 536150112

cat /sys/fs/cgroup/memory/docker/${CID}/memory.oom_control
# oom_kill_disable 0
# under_oom 0
# oom_kill 3

oom_kill: 3.

The kernel had killed a process in this container three times before tonight. Tonight was the fourth kill. The previous three happened during traffic peaks brief enough that the service restarted before the oncall alert threshold triggered. The host memory graph was flat through all three kills. The host had no problem. The container had a problem. Nothing was watching the container.

memory.oom_control is a file. It contains a counter. The counter is called oom_kill. It starts at zero when the cgroup is created. It increments every time the cgroup OOM killer fires. It should be zero. If it is not zero, processes in that container have been killed and the killer’s accounting proves it.

The fix

Raise the limit to something the service can actually use:

services:
  api-service:
    mem_limit: 1536m

Restart. The service came up, ran through startup, loaded the dataset, started serving traffic. The host had the RAM. The container had just never been given permission to use it.

This is a ten-second fix. The hard part is knowing you need it, which requires watching the right counter.

What to collect

Host memory monitoring tells you when the host is running out. It tells you nothing about what is happening inside containers. The two monitoring surfaces are separate by design.

Per-container memory monitoring needs:

• memory.usage_in_bytes / memory.limit_in_bytes as a ratio, alerted at 80%. Those two files exist for every Docker container under /sys/fs/cgroup/memory/docker/<id>/. Division. One alert per container.
• memory.oom_control’s oom_kill value as a counter, alerted on any nonzero value. A cgroup that has killed a process has a problem that needs investigating regardless of whether the service recovered.
• dmesg forwarded to wherever you collect logs, with filtering for Memory cgroup out of memory.

None of this requires additional software. The numbers are in files. The files are in /sys/fs/cgroup/memory/docker/<container-id>/. If you’re running cgroups v2 (most current kernels default to it now), the paths move to /sys/fs/cgroup/docker/<id>/ and the file is memory.events with an oom_kill field instead of memory.oom_control. Same counter, different path.

# cgroups v1
cat /sys/fs/cgroup/memory/docker/${CID}/memory.oom_control | grep oom_kill

# cgroups v2
cat /sys/fs/cgroup/docker/${CID}/memory.events | grep oom_kill

Check which version your kernel uses: stat /sys/fs/cgroup/cgroup.controllers exists on v2, doesn’t on v1.

The pattern

The disk with forty percent free. The listen queue at five since 2014. Now this: a limit set in a provisioning template, copied to a service it didn’t fit, left alone for eleven months, and then wrong in exactly the way nobody was watching for.

The OOM killer doing its job on the host is visible in host metrics. The cgroup OOM killer is not. The host was fine. The container had a ceiling and was hitting it repeatedly. The counter for that is oom_kill. It is a file. It is zero or it is not zero.

That counter had been nonzero for eleven months. Eleven months of dmesg entries, eleven months of killed processes, eleven months of restarts that recovered fast enough that the oncall threshold never triggered. The service was silently being murdered on a recurring basis and nobody knew because nobody was reading the right file.

memory.oom_control. Find it for every container you run in production. If oom_kill is not zero, find out why before the next 04:06 page.