$RodHat_/search
Rod's Tales

The memory was freed and the heap kept growing.

Published by

This article may contain Amazon affiliate links. I may earn a commission if you buy through them. Disclosure.

Editorial card: The memory was freed and the heap kept growing.
Editorial card: RodHat

Three weeks. That is how long we watched this service eat memory before we figured out where it was going.

It was a thumbnail generation daemon, written in C. Worker pool, job queue, the usual shape: pull an image off the queue, decode it with libpng or libjpeg depending on what you got, do the resize with libswscale, encode the output, write it to object storage, free everything, grab the next job. Eight worker threads, four cores, a box with 8GB of RAM and a healthy 4GB of swap to absorb the occasional spike.

RSS started around 600MB on restart. By the end of day one it was 900MB. By the end of day three it was 1.6GB. After ten days it was sitting around 3GB and climbing. At day eighteen the OOM killer came in and wrapped things up.


The first instinct when a service leaks memory is to look for the leak. That is the correct instinct. We ran it under valgrind. Valgrind, with full leak checking and every suppression file removed, reported zero leaks. Clean. “All heap blocks were freed — no leaks are possible.”

We added AddressSanitizer. Same result: clean shutdown, no leaked blocks.

We read the code again. Every malloc had a corresponding free on every exit path. Error paths, timeout paths, early returns. We traced them by hand. Nothing was missing.

We added instrumentation: a counter of outstanding allocations, incremented on malloc, decremented on free. We logged it every thirty seconds. It was stable. It never climbed. The number of live allocations at any given moment was roughly constant, which is what you’d expect from a worker pool churning through a steady job stream.

And yet the RSS kept growing.


After three days of this, I started looking at what memory the process actually had, rather than what it claimed to be using.

pmap -x on the running process:

$ pmap -x $(pgrep thumbnailerd) | sort -rn -k3 | head -20
...
00007f8a18000000   65536   62340   62340 rw---  [ anon ]
00007f8a14000000   65536   61020   61020 rw---  [ anon ]
00007f8a10000000   65536   59880   59880 rw---  [ anon ]
00007f8a0c000000   65536   58440   58440 rw---  [ anon ]
00007f8a08000000   65536   57220   57220 rw---  [ anon ]
00007f8a04000000   65536   56100   56100 rw---  [ anon ]
00007f8a00000000   65536   54960   54960 rw---  [ anon ]
...

Twenty-four anonymous 64MB mappings, all of them dirty, none of them named. The total was about 1.5GB of anonymous mappings that had nothing to do with the binary or its shared libraries.

Not the heap. The heap in glibc is usually a single mapping that grows with brk(). These were separate mmap() allocations, each 64MB, each looking like an independent heap.

They were. Each one was a glibc malloc arena.


glibc’s allocator does something that trips people up on multithreaded programs. A single heap protected by a mutex is a bottleneck: every thread that needs memory serializes on the same lock. To fix that, glibc allocates each thread its own arena, an independent heap with its own free lists and its own lock.

How many arenas? On a 64-bit system, the default limit is 8 * nproc. On our four-core box, that was thirty-two. The process had been running long enough that most of them had been populated.

Each arena is 64MB, grown in place from a mmap() call. When a thread frees memory, the freed blocks go into that thread’s arena’s free lists. The arena may or may not return memory to the OS depending on what the top chunk looks like.

The top chunk is the key. glibc calls munmap() to release an arena only if the top of the heap is one large contiguous free region. If you have a 64MB arena and 60MB of it is freed, but those freed blocks are interspersed with a handful of live allocations, glibc cannot release any of it. The live allocations are pinning the whole structure in place. The arena will happily return freed blocks to threads that ask for new allocations, but it will not give the memory back to the OS until the top chunk is clear.

Now multiply that behavior by thirty-two arenas, each with a slightly different fragmentation pattern, each pinned by a few live allocations scattered through its address space.

The arithmetic worked out. Thirty-two arenas at 64MB each is 2GB of address space, all of it technically available to malloc, none of it returning to the OS, all of it showing as RSS.


malloc_info(0, stdout) makes this visible. It prints an XML dump of every arena’s state: total size, free space, the distribution of free blocks by size class:

#include <malloc.h>
malloc_info(0, stdout);

The output for one representative arena:

<heap nr="4">
<sizes>
  <size from="33" to="48" total="192" count="4"/>
  <size from="49" to="64" total="576" count="9"/>
  <size from="65" to="80" total="1200" count="15"/>
  ...
  <unsorted from="131073" to="524288" total="47185920" count="3"/>
</sizes>
<total type="fast" count="28" size="1968"/>
<total type="rest" count="3" size="47185920"/>
<system type="current" size="67108864"/>
<system type="max" size="67108864"/>
<aspace type="total" size="67108864"/>
<aspace type="mprotect" size="67108864"/>
</heap>

Forty-seven megabytes of freed memory in the “rest” bins, waiting to be reused. Sixty-seven megabytes mapped from the OS. Twenty megabytes live, not a contiguous chunk at the top, so none of the forty-seven MB can go back. Multiply by twenty-four arenas and you have where the memory went.


The fix has two forms.

The first: MALLOC_ARENA_MAX=1 in the environment before the process starts. This caps the arena count at one, so all threads share a single heap. You reintroduce lock contention, which matters on workloads where allocations are on the critical path. For a thumbnail daemon spending most of its time in libpng and ffmpeg, the contention was not measurable.

MALLOC_ARENA_MAX=1 ./thumbnailerd

The second: mallopt(M_ARENA_MAX, 1) in the code, called before any threads are created. Same effect, without depending on the environment.

After either change, RSS stabilized around 700MB under full load and stayed there. The OOM kills stopped. The service ran for eleven weeks without a restart before we decided to take it down for an unrelated upgrade.


Valgrind cannot find this class of problem because it is not a bug in the conventional sense. You freed the memory. The allocator has it. The allocator is keeping it available for future allocations rather than returning it to the OS, because returning it to the OS is not free: it requires an munmap() call followed eventually by a page fault when the pages are needed again. glibc is making a reasonable tradeoff. The tradeoff is wrong for a long-running service on a memory-constrained box, but it is not a bug.

This is a different failure mode from what MADV_FREE and lazy page reclaim does to your RSS numbers, though the symptom looks similar. There, the allocator does return memory to the OS via madvise(MADV_FREE), but the kernel does not immediately reclaim it, so the pages stay in RSS until memory pressure forces them out. The memory is “freed” in the sense that the kernel can reclaim it, but the RSS counter stays high. With arena fragmentation you are not even getting to madvise: the memory is sitting in a free list inside the allocator, technically available, never going anywhere.

The OOM killer story from two months back is about a different failure: a genuine slow leak that evaded monitoring because the kernel kept killing the wrong process. That one required finding the actual allocation that was never freed. This one had no allocation that was never freed. All the allocations were freed. The problem was that “freed” and “returned to the OS” are not synonyms, and on a multithreaded process under glibc, they can diverge significantly.

If your long-running C or C++ daemon is growing in RSS and valgrind finds nothing, run malloc_info. Count your arenas. The linux-memory hub has the accounting background; this is what the RSS number looks like when the accounting is technically correct and still lying to you.

MALLOC_ARENA_MAX=1 is a one-line fix. I wish I had known it three weeks earlier.