$RodHat_/search
Console Tips

zpool iostat -l and -w: read pool latency, not just throughput

Published by

This article may contain Amazon affiliate links. I may earn a commission if you buy through them. Disclosure.

Server hardware used as a stand-in cover for ZFS pool diagnostics
On this page · 7 sections
  1. Start with -l and -v
  2. One bad disk in a mirror
  3. -w: histograms, because averages lie
  4. -q and -r: queues and request sizes
  5. Scripting it
  6. Related: zpool status -s
  7. Limits

“The storage is slow” is not a measurement. Most people reach for zpool iostat 1, see a few hundred MB/s and a few thousand ops, and conclude nothing, because bandwidth and IOPS were never the interesting numbers. Latency is. OpenZFS has shipped latency reporting in zpool iostat for years and it is still the least-used flag set on the box.

Start with -l and -v

zpool iostat -vl tank 5

-v breaks the pool into vdevs and leaf disks. -l appends average latency columns: total_wait and disk_wait for reads and writes, then syncq_wait, asyncq_wait, and scrub/trim wait. The distinction that matters:

  • disk_wait is time spent in the device itself.
  • total_wait is everything, including time queued inside ZFS before the device saw the I/O.
  • A big gap between the two means ZFS is queueing, not the disk being slow. Look at -q next.
  • syncq_wait high on writes is what an NFS server or a database with sync=standard feels as lag. asyncq_wait is background transaction group flushing and mostly harmless on its own.

Throw away the first report. Like iostat(8), the first block is the average since boot (since the pool was imported), which tells you about last Tuesday. Newer OpenZFS has -y to omit it. If your version lacks that, skip the first block by eye.

One bad disk in a mirror

This is the case where -v earns its keep. A mirror with a dying member keeps working; the pool average looks fine because reads get served by the healthy side and writes wait for the slow one. Per-leaf disk_wait shows it immediately: one disk at 40 ms writes while its partner sits at 2 ms. SMART may still say PASSED. Do not argue with the latency numbers; pull smartctl -a and check for reallocated and pending sectors, then zpool offline it before it takes the pool’s write speed down with it. Same instinct as in the storage was fine, we were swapping: confirm which layer is actually slow before replacing hardware.

-w: histograms, because averages lie

zpool iostat -w tank

-w prints a latency histogram with power-of-two buckets, per I/O class: total wait, disk wait, sync and async queue, scrub, trim. An average of 5 ms can be 99% at 200 microseconds and 1% at 400 ms, and that 1% is the one your users file tickets about. The histogram shows the tail. Add -v for per-vdev histograms. Counts are cumulative since import, so for a before-and-after, run it twice and compare, or restart your measurement window by looking at deltas rather than totals.

If you want a live view of the same thing from the block layer instead of from ZFS’s own accounting, bpftrace I/O latency gives you the device view to cross-check against.

-q and -r: queues and request sizes

zpool iostat -q tank 5
zpool iostat -r tank

-q shows queue depth per class: pending and active counts for sync read, sync write, async read, async write, scrub, trim. Persistent pending counts with low disk_wait means ZFS is holding I/O back (a scrub competing with production, for example) rather than the disk failing. -r gives a histogram of request sizes. If you tuned recordsize for a database and the histogram shows a wall of 128K writes anyway, your tuning did not land where you thought. Check the dataset property, then check what the application is actually issuing.

Scripting it

-H drops headers and uses tabs, -p prints exact numbers instead of rounded ones. Together with -T d for timestamps you can append to a file and graph later:

zpool iostat -Hpl -T d tank 10 >> /var/log/zpool-lat.log

Rotate that log. I have found a month of one-line-per-ten-seconds data in /var/log of a box whose actual problem was a full /var.

zpool status -s adds a SLOW column counting I/Os that exceeded zio_slow_io_ms. The default threshold is 30 seconds, which is generous to the point of being decorative, so treat a nonzero count as serious and a zero as uninformative. Lower the tunable if you want it to catch the 500 ms disks you actually care about.

Limits

  • Latency columns need a reasonably current OpenZFS. Older FreeBSD and Linux releases only give you the throughput view; check zpool(8) on your own box for which flags exist.
  • Averages and histograms are pool-level accounting. They cannot tell you which process or file caused the I/O. For that you need dtrace, or bpftrace on Linux, as in dtrace syscall latency.
  • ARC hits never touch the disk, so a cache-friendly workload will show low disk numbers while the application is still slow for other reasons. Read the ARC post before blaming the pool.

More of this lives in the ZFS and storage hub.