$RodHat_

Topic · 16 articles

Linux Memory

Your memory graph is lying to you, and it's lying in at least three different ways. RSS double-counts shared pages, allocators hand memory back lazily so freed pages still look resident, and the container limit you inherited from a template isn't on any host dashboard.

Read the accounting first, because every tuning decision downstream depends on reading the right number. Then the madvise family, then what it takes to keep a secret out of swap, core dumps, and the kernel's own direct map. The Tales are the bill for skipping the first part.

Start here

Read these first, in this order.

  1. 01

    Tip · 2026-08-20

    ps lies about memory. /proc/smaps_rollup does not.

    RSS from ps aux double-counts shared pages and makes every process look more expensive than it is. PSS from /proc/PID/smaps_rollup gives you actual per-process memory ownership. Here is how to read it.

  2. 02

    Tip · 2026-09-25

    MADV_FREE and the RSS number that's lying to you

    Since jemalloc 4.x, freed memory goes back to the OS with MADV_FREE instead of MADV_DONTNEED. Pages stay mapped, RSS stays inflated, and your monitoring is reading the wrong number. The fix is in /proc/pid/smaps_rollup.

  3. 03

    Tip · 2026-10-07

    process_madvise: memory pressure hints across process boundaries

    Linux 5.10 added process_madvise(2), the cross-process counterpart to madvise(2). You pass a pidfd and an iovec of address ranges in the target process and advise MADV_COLD, MADV_PAGEOUT, MADV_WILLNEED, or MADV_COLLAPSE. Android's LMKD uses it to reclaim memory before killing. You can use it to pre-freeze a process before CRIU checkpoint.

  4. 04

    Tip · 2026-09-16

    mlock, MADV_DONTDUMP, and where your secrets actually leak

    You zero the key buffer before free. The compiler deletes it. You call mlock but hit ENOMEM and ignore it. The coredump ships the private key to your crash reporter. Here is how memory hygiene for secret material actually works on Linux.

  5. 05

    Tip · 2026-09-18

    memfd_secret: memory the kernel itself cannot read

    mlock keeps secrets out of swap. memfd_secret goes further by removing the pages from the kernel's direct map entirely. No kernel module, no ptrace, no /proc/kcore can reach them. The tax is TLB pressure, which is why some distros ship it disabled.

  6. 06

    Tip · 2026-08-28

    userfaultfd: handle your own page faults from userspace

    Since Linux 4.3, userfaultfd() lets a dedicated handler thread intercept page faults on registered memory regions and supply the missing pages before the faulting thread resumes. CRIU uses it for post-copy restore. QEMU uses it for live migration. You can use it to build demand-paged memory backed by anything.

The syscalls underneath

Version introduced, sourced from the man page. Full cards in the syscall reference, or check your kernel.

InterfaceSinceWhat it does
madvise(2)4.4BSDAdvise the kernel how a memory range will be used: free it, page it out, exclude it from dumps.
process_madvise(2)Linux 5.10madvise() for another process's address space, addressed by pidfd and an iovec of ranges.
mlock(2)POSIX.1-1996 (SVr4)Lock pages into RAM so they can't be swapped out; mlockall() does the whole address space.
memfd_secret(2)Linux 5.14Create memory removed from the kernel's direct map, mapped only by fd holders.
memfd_create(2)Linux 3.17Create an anonymous RAM-backed file with no path, optionally sealable.
userfaultfd(2)Linux 4.3Create an fd that lets userspace resolve page faults on registered memory ranges.
pkey_alloc / pkey_mprotect(2)Linux 4.8Allocate a memory protection key and use it to gate access to a memory region via a per-thread register, without a syscall per permission switch.
rseq(2)Linux 4.18Register a per-thread struct that the kernel keeps updated with the current CPU ID, enabling lock-free per-CPU data updates via restartable critical sections.

More console tips (5)

Rod's tales (3)

MOTD (2)

Other topics