Somebody changes a sysctl, runs the benchmark once, sees 4% better, and writes it in the wiki as fact. I have done this. Run it again tomorrow and it is 3% worse, because the machine had a different mood. The answer to “is it actually faster” is statistics, and FreeBSD has shipped a tool for it in base for years. It is called ministat, it is one small C program, and almost nobody I meet has heard of it.
The basic form
You need two files, one number per line. Timings, throughput, latency, whatever you measured:
for i in $(jot 20); do
/usr/bin/time -f %e ./bench 2>&1 >/dev/null
done > before.txt
Change the one thing you are testing, repeat into after.txt, then:
ministat before.txt after.txt
ministat(1) draws a crude ASCII plot of both distributions, prints a table with N, min, max, median, average and standard deviation for each file, and then gives a verdict. The verdict is one of two lines. Either it reports a difference at the confidence level, with the size of the change and its error margin, or it says no difference was proven at 95.0% confidence. The second line is the one that saves you from yourself.
Read it literally. “No difference proven” does not mean “no difference exists”. It means your data cannot tell the two apart, and you probably need more runs or a quieter machine.
Flags worth knowing
-csets the confidence level. The man page lists a fixed set of values (80, 90, 95, 98, 99 and 99.5), not arbitrary numbers. Checkministat(1)on your release before you script it.-sprints the standard deviation line under each plot. Useful when the averages look close but one set is all over the place.-nsuppresses the plot and gives only the summary, which is what you want in a script.-wsets the width of the plot.-Aprints just the statistics and verdict, no plot.- With no file argument it reads standard input, so
ministatworks at the end of a pipe for a single set.
The option list has moved around a little between releases, so run ministat -h rather than trusting my memory of which letter does what.
Why the plot matters
Look at it before you read the verdict. Two clumps that barely overlap are a real change. A smear with the averages 2% apart is noise wearing a number. A bimodal pile (two clusters in one file) means something else was going on during your runs, usually a cron job, a cold cache on the first few iterations, or thermal throttling. The t-test assumes a roughly sane distribution; the plot is how you notice that it does not have one.
This is the same instinct as watching zpool iostat latency histograms instead of the average. A mean hides the shape.
Getting honest numbers into it
The tool cannot fix a bad experiment. A few rules I have paid for:
- Throw away the first run or two. Cold caches skew the head of the file. Better, prime the cache the same way every time.
- Interleave if you can. Run before, after, before, after. If you do all of one then all of the other, drift in the machine’s state (a scrub starting, a neighbour jail waking up) lands entirely on one side.
- Change one thing. If you edit two tunables, you get a verdict on the pair. Read sysctl, loader.conf and kenv to be sure which one actually applied. Plenty of boot-time tunables silently do nothing after boot.
- Take at least 10 samples, 20 is better. Five runs with a noisy workload will rarely prove anything.
If you are chasing why something is slow rather than whether a change helped, you want perf stat and record or dtrace first, and ministat second as the referee.
Not on FreeBSD
ministat is not in Linux coreutils. Some distributions package it separately, and the source is small enough to build by hand; check your package manager rather than assuming. If it is not available, the principle still holds: any two-sample t-test in whatever you have beats eyeballing one run.
I used to distrust anything that boiled my benchmark down to a single sentence. Then I watched it say “no difference proven” about a tunable I had been recommending for a year. Grudging respect. More in the FreeBSD and OpenBSD hub.
