Every few months someone shows me a forty-line Python script that answers “which hosts are in the inventory but not in monitoring.” It loads two files into sets and subtracts them. It is correct. It is also comm -23, which has been in POSIX since before that person’s parents met.
What comm does
comm reads two sorted files and prints three columns: lines only in the first, lines only in the second, lines in both. You suppress columns with -1, -2, -3.
sort -u inventory.txt > /tmp/a
sort -u monitoring.txt > /tmp/b
comm -23 /tmp/a /tmp/b # in inventory, missing from monitoring
comm -13 /tmp/a /tmp/b # in monitoring, not in inventory (stale alerts)
comm -12 /tmp/a /tmp/b # intersection
Read the flags as “suppress these columns.” -23 hides columns two and three, leaving only what is unique to file one. Once you internalize that, set difference, reverse difference and intersection are three flag combinations and no code.
The sorted-input contract (where it lies to you)
comm does a single merge pass. It assumes both inputs are sorted in the same collation order it will use. Feed it unsorted input and it does not error politely. GNU comm warns that input is not in sorted order; other implementations may just emit garbage. A script that ignores stderr gets a wrong answer and a zero exit.
The nastier version is a locale mismatch. You sort with one locale and compare with another, or your cron job runs with LANG=C while your terminal runs en_US.UTF-8. The two orderings disagree about case and punctuation, and comm decides some lines are unique when they are not. The fix is to pin the locale on every command in the chain:
export LC_ALL=C
sort -u a.txt > a.s
sort -u b.txt > b.s
comm -23 a.s b.s
Byte order is boring and identical everywhere. Boring is the feature. If you need sort to use a different order, pass the same order to comm and make sure you tested it with mixed-case data, because that is where it breaks.
You can skip the temp files with process substitution in bash, ksh93 or zsh:
comm -23 <(sort -u a.txt) <(sort -u b.txt)
Plain POSIX sh has no <(...), so on a bare FreeBSD /bin/sh script you write the temp files and clean them up with a trap. I covered that pattern in the trap post.
join: the same idea with keys
join is comm with columns. Two files sorted on a join field, matched on that field, output merged. It is a relational inner join from 1970s tooling.
# hosts.txt: host owner
# disks.txt: host size
sort -k1,1 hosts.txt > h.s
sort -k1,1 disks.txt > d.s
join h.s d.s
Useful flags: -1 N and -2 N pick the key field in each file, -t , sets the delimiter (the default is runs of blanks), -a 1 also prints unmatched lines from file one (a left outer join), and -v 1 prints only the unmatched lines from file one, which is an anti-join. The -a and -v options are in POSIX, so they work on FreeBSD and Linux alike.
The same locale rule applies, doubly. sort -k1,1 and join must agree on collation, so keep LC_ALL=C exported for the whole pipeline.
The cons
- Both tools need sorted input. For a giant unsorted stream you pay for a sort, which spills to disk. That is fine, but it is not free.
joinhandles one key field per side. Composite keys mean building a synthetic key column withawkfirst.- Duplicate keys produce a cross product of matching lines. If
hosts.txtlists a host twice, your output doubles and nobody warns you. Runsort -ufirst, or check withuniq -d. - Trailing whitespace and CRLF line endings make identical-looking lines differ. Strip with
tr -d '\r'before you blame the tool.
When to reach for awk instead
If the files are unsorted, huge and you only need membership, an awk associative array is one pass and no sort:
awk 'NR==FNR { seen[$0]; next } !($0 in seen)' b.txt a.txt
That prints lines in a.txt absent from b.txt, preserving original order, which comm cannot do. Pick comm when you want sorted output and tiny scripts, awk when order or memory-resident speed matters.
Plenty more of this kind of thing lives in shell one-liners worth screenshotting. Learn the old tools properly. They were right in 1979 and they are still right, provided you sort first and pin the locale.
