cachestat(2): querying the page cache without mmap

mincore(2) has existed since Linux 2.3.99 and it has been the wrong tool for this job the entire time. You want to know how many of a file’s pages are resident? First mmap the file. Then call mincore. It returns a byte vector, one byte per page, low bit set if resident. That is it. No dirty count. No writeback state. No eviction statistics. And you had to mmap a potentially multi-gigabyte file to get there, which itself pollutes the cache you were trying to inspect.

Linux 6.5 added cachestat(2). Pass an fd and a byte range. Get back actual numbers. No mmap required.

The structs

#include <linux/mman.h>   /* struct cachestat_range, struct cachestat */
#include <sys/syscall.h>
#include <unistd.h>

struct cachestat_range {
    __u64 off;   /* start of range in bytes */
    __u64 len;   /* length in bytes; 0 means to EOF */
};

struct cachestat {
    __u64 nr_cache;            /* pages currently in page cache */
    __u64 nr_dirty;            /* dirty pages not yet written out */
    __u64 nr_writeback;        /* pages under active writeback right now */
    __u64 nr_evicted;          /* pages evicted since last call */
    __u64 nr_recently_evicted; /* evicted under memory pressure since last call */
};

No glibc wrapper in anything you likely have. Use syscall(2) directly. SYS_cachestat is 451 on x86-64. The flags argument must be zero.

long ret = syscall(SYS_cachestat, fd, &range, &cs, 0);
if (ret < 0)
    err(1, "cachestat");

Checking if a file is hot before a benchmark

The canonical use: you want to verify the kernel actually has your dataset warm before you start a benchmark, without influencing the cache state by reading the file yourself.

#include <fcntl.h>
#include <err.h>
#include <stdio.h>
#include <linux/mman.h>
#include <sys/syscall.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    if (argc < 2) errx(1, "usage: cachewarm <file>");

    int fd = open(argv[1], O_RDONLY | O_CLOEXEC);
    if (fd < 0) err(1, "open");

    struct cachestat_range range = { .off = 0, .len = 0 }; /* whole file */
    struct cachestat cs = {0};

    if (syscall(SYS_cachestat, fd, &range, &cs, 0) < 0)
        err(1, "cachestat");

    /* page size is 4096 on x86; use sysconf(_SC_PAGESIZE) to be portable */
    long pgsz = sysconf(_SC_PAGESIZE);

    printf("resident:  %llu pages (%llu MiB)\n",
           cs.nr_cache, cs.nr_cache * pgsz / (1024*1024));
    printf("dirty:     %llu pages\n", cs.nr_dirty);
    printf("writeback: %llu pages\n", cs.nr_writeback);
    printf("evicted:   %llu pages (since last call)\n", cs.nr_evicted);

    close(fd);
    return 0;
}

If nr_cache is less than file_size / page_size, the file is not fully resident. Run posix_fadvise(fd, 0, 0, POSIX_FADV_WILLNEED) and call cachestat again after a second. If it is still short, you have a cache pressure problem, not a warmup problem.

Dirty pages before a hard shutdown

nr_dirty tells you how many pages have been written to but not yet flushed. On a database or write-heavy workload, this is the number that matters before you pull power. Call sync(2) and poll until nr_dirty drops to zero, or accept that you are relying on the storage device’s write buffer.

/* wait for dirty pages on a specific fd to flush */
for (;;) {
    if (syscall(SYS_cachestat, fd, &range, &cs, 0) < 0)
        err(1, "cachestat");
    if (cs.nr_dirty == 0 && cs.nr_writeback == 0)
        break;
    usleep(100000); /* 100ms */
}

This is more precise than watching /proc/meminfo’s Dirty: field, which is global. You are watching one fd.

What nr_evicted and nr_recently_evicted mean

Both counts reset relative to the previous cachestat call on the same fd. nr_evicted counts pages that were in cache and got reclaimed between calls. nr_recently_evicted counts pages reclaimed under memory pressure (the kernel marks these with PG_workingset before dropping them, so the counts differ when the eviction is by explicit MADV_FREE or MADV_DONTNEED versus kernel reclaim). A non-zero nr_recently_evicted on a hot file means you are losing working set to memory pressure, which is the honest diagnosis for “why is my database doing so much read I/O when the data should fit in RAM.”

Scope and caveats

cachestat only covers the page cache. Hugepage-backed files (MAP_HUGETLB), DAX mappings, and tmpfs on a ramfs mount report zeros or partial data. The call is unprivileged: any process that can open the file can query its cache state. That is intentional.

Kernel 6.5+. Check before deploying:

uname -r   # want 6.5 or later

On older kernels, fall back to mincore(2) for the resident count and accept that you cannot get dirty/writeback data without digging into /proc/<pid>/smaps or /proc/meminfo for system-wide approximations. See smaps and PSS for what that looks like.

BSD

FreeBSD has no equivalent syscall. The BSD VFS does not expose per-file page cache state at this level without a kernel module. mincore(3) is there, with the same “mmap first” requirement. If you are on FreeBSD and you need this information for a hot data path, madvise(2) hints (MADV_WILLNEED) are the best you can do short of a custom kmod. It is one of the few places where the Linux interface is genuinely ahead.