Someone says “the box is slow” and the first move on Linux is iostat -x. FreeBSD has iostat too, but gstat(8) is the one I leave running in a spare tmux pane, because it reads the GEOM statistics live and refreshes like top for storage.
Run it like this
gstat -dp -I 1s
-p shows only physical providers (the actual disks), which strips out the partition, label, and mirror layers that otherwise triple the screen. -d adds the delete (TRIM) columns. -I 1s sets the refresh interval. Without -p you see every GEOM provider on the machine, and on a ZFS box with GPT labels that is a wall of noise.
The columns
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name
0 412 380 48320 0.4 32 512 1.1 0 0 0.0 18.2 ada0
- L(q): queue length. Requests handed to the device and not yet finished.
- ops/s, r/s, w/s: operations per second, total and split.
- kBps: throughput, per direction.
- ms/r, ms/w, ms/d: average latency per read, write, and delete. This is the column that matters.
- %busy: the share of the sample interval during which at least one request was outstanding.
Read ms/r and ms/w first. A spinning disk doing 8 to 12 ms per random read is healthy. The same disk showing 200 ms on writes while its siblings in the mirror show 10 is a drive on its way out, and gstat will show it long before smartctl admits anything.
Why %busy over 100 is not a bug
On an old single-spindle disk, %busy near 100 means saturated, because it handles one thing at a time. On an SSD or NVMe device, many requests are in flight at once. %busy only says “something was outstanding,” so it pegs at 100 and can read above it, while the device has plenty of headroom. gstat(8) says as much in its own man page.
So for flash, ignore %busy. Watch L(q) and the latency columns. A flash device with a queue length climbing and ms/w rising is saturated. One sitting at 100% busy with 0.2 ms latency is fine and just bored.
Finding the one bad disk in a pool
This is the use I care about. A ZFS mirror or raidz is only as fast as its slowest member, and the slow member rarely announces itself. Run gstat -p while the pool is under load, and compare the latency column across the vdev members:
gstat -p -f 'ada[0-3]$' -I 500ms
-f takes a regular expression matched against the provider name. The $ keeps ada1 from also matching ada10. One disk with several times the ms/w of its peers, and a deeper L(q), is your suspect. Cross-check with per-vdev numbers from zpool iostat latency histograms: ZFS tells you which vdev is slow, gstat tells you which physical device under it is the cause.
Reading it next to something else
gstat shows you the device, not the process. When the device is hot and you need to know who is doing it, top -m io gives per-process I/O, and the procstat kernel stack trick will tell you what a stuck process is waiting on. On the Linux side the equivalent question gets answered with bpftrace block I/O latency.
Where it bites
- It averages over the interval. A 5 ms stall inside a 1 second window is invisible. For spikes, shorten
-Ior use DTrace. - Layers double count. Without
-p, a write to a ZFS-on-GPT disk appears on the partition and the disk. Do not add them up. - It only knows GEOM. Network storage and anything not behind a GEOM provider will not show up here.
- Device names move.
ada0today can beada1after a cable swap. Useglabelorgpartlabels inzpool statusif you want a name you can trust, and checkgstat(8)for the flags on your release before scripting around the output. Batch mode exists for that, and the man page is shorter than this post.
I distrusted gstat for years because “it’s just a pretty top.” Then it caught a mirror member eating 300 ms writes while SMART reported the drive as healthy, and I stopped being clever about it. Irritating, as usual.
More in the FreeBSD and OpenBSD hub and the ZFS and storage hub.
