No, ZFS did not eat your RAM. Read arc_summary before you tune anything.

Every few months somebody discovers their ZFS box is “using 90% of its RAM” and reaches for zfs_arc_max. Usually they should not. Occasionally they absolutely should. Telling those two apart takes one command and about four minutes.

What the ARC is

The Adaptive Replacement Cache is ZFS’s own page cache, and it is not the kernel’s. That’s the important part. On Linux, ZFS came from Solaris and brought its cache with it rather than integrating with the standard page cache, which is why the ARC shows up in memory accounting as something scary rather than as ordinary reclaimable cache.

It’s deliberately greedy: unused RAM is wasted RAM, so the ARC grows to fill it and shrinks when something else needs memory. “90% used” is the design working. The failure mode is not that it’s big — it’s that under sudden pressure it can shrink slower than an allocator needs it to, and the thing that wanted the memory gets OOM-killed while a hundred gigabytes of evictable cache sits there.

The command

arc_summary          # Linux (zfs-utils) and FreeBSD both ship it

Skip to the top block:

ARC size (current):                    78.4 %   94.1 GiB
        Target size (adaptive):        79.1 %   94.9 GiB
        Min size (hard limit):         6.2 %     7.5 GiB
        Max size (high water):         16:1     120.0 GiB

ARC hash breakdown:
        Elements max:                            8.2M
ARC total accesses:                              1.4G
        Cache hit ratio:                92.1 %   1.3G
        Cache miss ratio:                7.9 %   112.4M

Three numbers decide everything:

Current vs. Target. If current is at target, the ARC is where it wants to be. If current is well below target, something is applying memory pressure and the ARC is being evicted — that’s your signal that RAM is genuinely contended, not that ZFS is hoarding.

Hit ratio. Above ~90% on a read-heavy workload means the cache is earning its RAM. If you cap it, those hits become disk I/O. Below ~60% and the ARC is holding data nobody reads twice — capping it costs you very little.

MFU vs. MRU further down the output. The ARC’s whole trick over a plain LRU is tracking both recently-used and frequently-used data, so a single big sequential scan can’t flush your working set. If MFU dominates, you have a real working set being served from RAM. If MRU dominates, you’re mostly streaming, and a big ARC is doing less than the number suggests.

When to actually cap it

Two cases, and only two.

You’re running a database that has its own buffer cache. PostgreSQL’s shared_buffers, MySQL’s InnoDB buffer pool, and the ARC will all cache the same blocks. Now you’re paying twice in RAM for one copy of the data, and the database’s cache is smarter about its own access patterns than ZFS can be. Cap the ARC, give the memory to the database, and set primarycache=metadata on the dataset holding the data files so ZFS caches the structure but not the pages the DB is already caching.

Something with a spiky allocation pattern is getting OOM-killed. The ARC shrinks on pressure, but a JVM taking a 30GB heap in one go can lose the race. If your kernel log shows OOM kills while arc_summary shows a fat ARC, cap it and leave headroom.

Neither case is “the number looked big in free -h.”

How to cap it, correctly

Linux, live and persistent:

echo $((64 * 1024**3)) > /sys/module/zfs/parameters/zfs_arc_max
echo "options zfs zfs_arc_max=68719476736" > /etc/modprobe.d/zfs.conf

FreeBSD:

sysctl vfs.zfs.arc_max=68719476736
echo 'vfs.zfs.arc.max="68719476736"' >> /boot/loader.conf

Set zfs_arc_min too, or under pressure the ARC can collapse to almost nothing and your hit ratio falls off a cliff at exactly the moment the box is already struggling. A floor of a quarter of your max is a reasonable starting point.

Then watch the hit ratio for a week. If it dropped ten points and your disk read IOPS doubled, you traded RAM you weren’t using for I/O you now pay for on every request. Put it back.

The one that actually bites

arc_summary also reports the metadata cache separately. On a pool with tens of millions of small files — a mail spool, a build cache, anything with a deep tree — you can be metadata-starved while the data cache looks healthy, and every ls walks the disk. zfs_arc_meta_limit (or a dedicated special vdev on real hardware) is the lever there, and it’s a completely different problem from the one everyone thinks they have when they see a big number in free.

Read the summary. Then tune. In that order — the other order is how you turn a working cache into a mystery.