awk has hash maps, and that's why you don't need the Python script

There’s a moment in every incident where somebody says “let me write a quick script” and disappears for twenty minutes. Nine times out of ten the question was “which endpoint is generating the errors” and the answer was one line of awk.

awk is a real language with hash maps in it. That’s the part people miss.

The pattern

{ count[$7]++ } END { for (k in count) print count[k], k }

Three ideas: an associative array indexed by anything, auto-vivification (no declaration, no initialization — referencing count[$7] creates it at zero), and an END block that runs after the last line. Every log question is a variation.

Top URLs in an access log:

awk '{c[$7]++} END{for(k in c) print c[k], k}' access.log | sort -rn | head -20

Bytes per client:

awk '{b[$1]+=$10} END{for(k in b) printf "%12.1f MB  %s\n", b[k]/1048576, k}' access.log | sort -rn

Status code breakdown per endpoint — two-dimensional, using awk’s subscript separator:

awk '{c[$7,$9]++} END{for(k in c){split(k,p,SUBSEP); print p[1], p[2], c[k]}}' access.log

That’s a GROUP BY on two columns in a language that shipped in 1977 and is installed on literally every Unix machine you will ever touch, including the busybox one in the container you can’t rebuild.

The one that actually matters in an incident

Averages hide everything, so compute the distribution:

awk '{
  n[$7]++; s[$7]+=$11
  if ($11 > mx[$7]) mx[$7] = $11
}
END {
  printf "%-40s %8s %10s %10s\n", "endpoint", "count", "avg_ms", "max_ms"
  for (k in n) printf "%-40s %8d %10.1f %10.1f\n", k, n[k], s[k]/n[k], mx[k]
}' access.log | sort -k2 -rn | head

Count, mean, and max per endpoint, formatted, in eight lines. The max column is the one that finds the problem — an endpoint with a 40ms mean and a 31000ms max is where your p99 lives, and no dashboard you own is going to volunteer that at 3am.

Real percentiles need the values retained, which awk can do but starts to get ugly. That’s the honest line where you reach for something else — though sort -n | awk 'NR==int(n*0.99)' gets you there if you’re stubborn, and I usually am.

Rate over time

Bucket by minute and you have a time series:

awk '{split($4,t,":"); m=t[2]":"t[3]; c[m]++} END{for(k in c) print k, c[k]}' access.log | sort

Errors only, per minute, so you can see the shape of the incident:

awk '$9 >= 500 {split($4,t,":"); c[t[2]":"t[3]]++} END{for(k in c) print k, c[k]}' access.log | sort

That leading $9 >= 500 is a pattern — awk’s core structure is pattern { action }, and if you write the pattern you don’t need a grep in front. It also means you’re comparing numerically, so $9 >= 500 does the right thing where grep ' 5.. ' matches a byte count that happens to look like a status code.

Multi-file and field separators

awk -F: '{print $1, $7}' /etc/passwd
awk -F'\t' '{...}' data.tsv
awk 'FNR==1 {print "== " FILENAME}' *.log        # FNR resets per file, NR doesn't
awk 'NR==FNR {seen[$1]; next} $1 in seen' ids.txt events.log   # join two files

That last one is worth memorizing. NR==FNR is true only while reading the first file — so the first block builds a set from ids.txt, next skips the rest, and once the second file starts, NR!=FNR and the last pattern filters events.log to lines whose first field was in the set. That’s a semi-join, in about forty characters, with no database.

When to stop

awk is right when the input is line-oriented, roughly columnar, and the question is a count, a sum, or a filter. It stops being right the moment you need JSON (use jq), nested structure, real floating-point care, or anything you’ll have to maintain for a year — awk’s readability curve goes vertical fast, and a 30-line awk program is a liability with nobody to inherit it.

But the twenty-minute Python script that gets thrown away after the incident? That was one line, and you’d have had the answer before the meeting started.