$RodHat_
Console Tips

mlock, MADV_DONTDUMP, and where your secrets actually leak

Published by

Editorial card: mlock, MADV_DONTDUMP, and where your secrets actually leak
Photo: RodHat / RodHat Editorial

There are three ways your program leaks key material without you knowing. The first is swap. The second is coredumps. The third is fork. Most programs that handle secrets get at least two of these wrong, and the failure is silent: no crash, no warning, just your private key sitting in a swap file on the disk.

Here is how to fix all three.

Why mlock fails in practice

mlock(addr, len) pins specific pages in RAM. mlockall(MCL_CURRENT | MCL_FUTURE) locks the entire process address space, current and future allocations. For anything that handles long-lived key material, mlockall is the easier call.

if (mlockall(MCL_CURRENT | MCL_FUTURE) < 0)
    err(1, "mlockall");

You will get ENOMEM on most systems unless you raise the limit first. The default RLIMIT_MEMLOCK on Linux is 64KB per process, which is enough for a handful of key-sized buffers but not a whole program. Options:

  • setrlimit(RLIMIT_MEMLOCK, &r) in code before the mlockall call
  • LimitMEMLOCK=infinity in a systemd unit file
  • memlock unlimited in /etc/security/limits.conf
  • CAP_IPC_LOCK grants mlockall without limits for the process

The wrong way to handle this: call mlockall, check errno, log a warning, continue anyway. That is almost every program that “supports” memory locking. The call failed. The pages are swappable. The warning goes to a log nobody reads. The key ends up in swap six weeks later during a memory crunch at 3am.

Either fail hard on ENOMEM or do not bother calling mlockall. There is no useful middle ground.

If mlockall is too aggressive for a process with a large heap, pin only the sensitive buffers. Allocate with posix_memalign to get page alignment that mlock requires, then lock exactly what you need:

unsigned char *key;
size_t len = 32;  /* AES-256 key */

posix_memalign((void **)&key, getpagesize(), len);
mlock(key, len);

mlock with a non-page-aligned address on older kernels rounds down silently, which means you might not be locking what you think. posix_memalign removes the ambiguity.

Coredumps

mlock does nothing about coredumps. A process that crashes dumps everything it has, including every locked page. If you have crash reporting that ships coredumps somewhere, you are shipping your keys.

madvise(MADV_DONTDUMP) tells the kernel to skip that region when writing a core file:

madvise(key, len, MADV_DONTDUMP);

Available since Linux 3.4. The complementary call is MADV_DODUMP, which re-includes a region if you marked a larger range with DONTDUMP but want specific pages back in.

For coarser control, /proc/self/coredump_filter is a bitmask of segment types that get included in core files. The bits:

  • bit 0: anonymous private mappings
  • bit 1: anonymous shared mappings
  • bit 2: file-backed private mappings
  • bit 3: file-backed shared mappings
  • bit 4: ELF headers
  • bit 5: private huge pages
  • bit 6: shared huge pages
  • bit 7: private DAX pages
  • bit 8: shared DAX pages

The default on most systems is 0x33 (anonymous private, file-backed private, ELF headers, private huge pages). A server that has no business generating useful coredumps in production can set it to 0x00 at startup:

int fd = open("/proc/self/coredump_filter", O_WRONLY);
write(fd, "0x00\n", 5);
close(fd);

Per-buffer MADV_DONTDUMP is the right tool when you need coredumps for debugging but want specific regions excluded. The filter is the right tool when you want no coredumps at all.

Fork and the forked child

fork() copies the parent’s address space. Every key in the parent is now in the child. If that child is going to exec into something else, the key material sits in its heap until the new program overwrites it, and the new program has no idea it is there.

madvise(MADV_DONTFORK) prevents the marked region from appearing in the child at all:

madvise(key, len, MADV_DONTFORK);

The problem: if anything in the child reads from a DONTFORK page before exec, it gets SIGBUS. That includes signal handlers, certain libc internals, and any library code that runs in the gap between fork and exec. You need high confidence that nothing touches those pages in the child before exec. That confidence is often not justified.

Linux 4.14 added the better answer: MADV_WIPEONFORK.

madvise(key, len, MADV_WIPEONFORK);

The child gets the pages. They are zero-filled. The parent is unchanged. No SIGBUS. The child cannot read the key material because there is no key material, just zeros. If something in the child reads the region before exec, it reads zeros and continues normally.

MADV_WIPEONFORK is correct for most use cases. MADV_DONTFORK is correct when you cannot tolerate even a zero-filled region in the child at all. In practice, WIPEONFORK is safer and simpler.

The memset problem

When you are done with a key, you zero it out. You have been doing this. You are still getting it wrong.

/* The compiler removes this. key is dead after the memset. */
memset(key, 0, len);
free(key);

This is legal C. The compiler sees that key is not read after the memset. Dead store elimination removes the call. The key bytes are still in memory when free runs, and when the next allocation reuses that block, they are still there.

OpenSSL had this bug. GnuTLS had this bug. Half the TLS stacks that existed before 2012 had this bug. The compiler was not wrong. The programmer was trusting the wrong abstraction.

The fix is explicit_bzero:

explicit_bzero(key, len);
free(key);

explicit_bzero is in glibc 2.17, musl, and every BSD libc. It is a memset that the standard and the compiler are prohibited from optimizing away. On platforms without it, memset_s from C11 Annex K is the equivalent, though memset_s has essentially no adoption in Linux libc implementations so you will probably not have it. OPENSSL_cleanse works if you are already in the OpenSSL ecosystem. All three mean “zero this and do not delete the instruction.”

Do not roll your own with a volatile pointer loop unless you have no other option. The existing functions are correct. The custom ones are usually subtly wrong.

The complete pattern for a secret buffer

#include <sys/mman.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>

unsigned char *alloc_secret(size_t len) {
    unsigned char *buf;

    if (posix_memalign((void **)&buf, getpagesize(), len) != 0)
        return NULL;

    mlock(buf, len);
    madvise(buf, len, MADV_DONTDUMP);
    madvise(buf, len, MADV_WIPEONFORK);

    return buf;
}

void free_secret(unsigned char *buf, size_t len) {
    explicit_bzero(buf, len);
    munlock(buf, len);
    free(buf);
}

Call mlockall(MCL_CURRENT | MCL_FUTURE) at process start if the whole program deals in key material. Use alloc_secret / free_secret for any buffer holding a private key, password, session token, or anything else you would not want in a coredump. Set the coredump filter to 0x00 if you are running in production and will never usefully analyze a core file anyway.

None of this defeats an attacker with root who can read /proc/pid/mem. That is a different problem, and the answers are hardware enclaves and TPMs, not madvise flags. What this defends against: crash-reporter exfiltration, swap-file forensics on a decommissioned disk, and accidental key inheritance across a fork into an unrelated child process. Those are the threats that actually show up in postmortems.

For generating the key material that goes into these buffers, the getrandom and vDSO post covers why you call getrandom(2) directly instead of reading /dev/urandom through a file descriptor.

Seccomp-bpf compounds this defense: if the process is sandboxed and cannot call ptrace or process_vm_readv, an attacker who exploits a bug in the process still cannot easily extract keys. The seccomp-bpf post covers the filter setup.

The memfd_create post is relevant if you are passing key material between processes via file descriptor rather than shared memory, which avoids the fork/exec inheritance problem entirely.