$RodHat_/search
Console Tips

comm and join are set operations you already own

Published by

This article may contain Amazon affiliate links. I may earn a commission if you buy through them. Disclosure.

Terminal comparing two sorted lists with comm
AI illustration: RodHat
On this page · 5 sections
  1. What comm does
  2. The sorted-input contract (where it lies to you)
  3. join: the same idea with keys
  4. The cons
  5. When to reach for awk instead

Every few months someone shows me a forty-line Python script that answers “which hosts are in the inventory but not in monitoring.” It loads two files into sets and subtracts them. It is correct. It is also comm -23, which has been in POSIX since before that person’s parents met.

What comm does

comm reads two sorted files and prints three columns: lines only in the first, lines only in the second, lines in both. You suppress columns with -1, -2, -3.

sort -u inventory.txt  > /tmp/a
sort -u monitoring.txt > /tmp/b

comm -23 /tmp/a /tmp/b   # in inventory, missing from monitoring
comm -13 /tmp/a /tmp/b   # in monitoring, not in inventory (stale alerts)
comm -12 /tmp/a /tmp/b   # intersection

Read the flags as “suppress these columns.” -23 hides columns two and three, leaving only what is unique to file one. Once you internalize that, set difference, reverse difference and intersection are three flag combinations and no code.

The sorted-input contract (where it lies to you)

comm does a single merge pass. It assumes both inputs are sorted in the same collation order it will use. Feed it unsorted input and it does not error politely. GNU comm warns that input is not in sorted order; other implementations may just emit garbage. A script that ignores stderr gets a wrong answer and a zero exit.

The nastier version is a locale mismatch. You sort with one locale and compare with another, or your cron job runs with LANG=C while your terminal runs en_US.UTF-8. The two orderings disagree about case and punctuation, and comm decides some lines are unique when they are not. The fix is to pin the locale on every command in the chain:

export LC_ALL=C
sort -u a.txt > a.s
sort -u b.txt > b.s
comm -23 a.s b.s

Byte order is boring and identical everywhere. Boring is the feature. If you need sort to use a different order, pass the same order to comm and make sure you tested it with mixed-case data, because that is where it breaks.

You can skip the temp files with process substitution in bash, ksh93 or zsh:

comm -23 <(sort -u a.txt) <(sort -u b.txt)

Plain POSIX sh has no <(...), so on a bare FreeBSD /bin/sh script you write the temp files and clean them up with a trap. I covered that pattern in the trap post.

join: the same idea with keys

join is comm with columns. Two files sorted on a join field, matched on that field, output merged. It is a relational inner join from 1970s tooling.

# hosts.txt:  host owner
# disks.txt:  host size
sort -k1,1 hosts.txt > h.s
sort -k1,1 disks.txt > d.s
join h.s d.s

Useful flags: -1 N and -2 N pick the key field in each file, -t , sets the delimiter (the default is runs of blanks), -a 1 also prints unmatched lines from file one (a left outer join), and -v 1 prints only the unmatched lines from file one, which is an anti-join. The -a and -v options are in POSIX, so they work on FreeBSD and Linux alike.

The same locale rule applies, doubly. sort -k1,1 and join must agree on collation, so keep LC_ALL=C exported for the whole pipeline.

The cons

  • Both tools need sorted input. For a giant unsorted stream you pay for a sort, which spills to disk. That is fine, but it is not free.
  • join handles one key field per side. Composite keys mean building a synthetic key column with awk first.
  • Duplicate keys produce a cross product of matching lines. If hosts.txt lists a host twice, your output doubles and nobody warns you. Run sort -u first, or check with uniq -d.
  • Trailing whitespace and CRLF line endings make identical-looking lines differ. Strip with tr -d '\r' before you blame the tool.

When to reach for awk instead

If the files are unsorted, huge and you only need membership, an awk associative array is one pass and no sort:

awk 'NR==FNR { seen[$0]; next } !($0 in seen)' b.txt a.txt

That prints lines in a.txt absent from b.txt, preserving original order, which comm cannot do. Pick comm when you want sorted output and tiny scripts, awk when order or memory-resident speed matters.

Plenty more of this kind of thing lives in shell one-liners worth screenshotting. Learn the old tools properly. They were right in 1979 and they are still right, provided you sort first and pin the locale.