Topic
Unix Shell & Tooling
Shell techniques, Unix utilities, process inspection, automation, and tools worth remembering.
Your process doesn't need 400 syscalls. seccomp-BPF lets you say so.
seccomp-BPF loads a classic BPF filter into the kernel that runs on every syscall your process makes. Give it a whitelist, everything else gets EPERM or SIGKILL. Here's how the filter model works, how libseccomp makes it tolerable, and how to audit what a real binary actually needs.
cgroups v2 is just files. Here's how to use them without touching systemd.
Linux cgroups v2 exposes memory, CPU, and I/O limits through a plain filesystem interface under /sys/fs/cgroup/. No systemd, no container runtime — write to files, read the results. Here's the real workflow for memory.max, cpu.max, freezer, and io.max.
ftrace has been on your machine since 2.6.27. Here's how to actually use it.
ftrace is Linux's built-in kernel function tracer, accessible directly via /sys/kernel/debug/tracing/. No compiler, no LLVM, no kernel headers — write to files, read call graphs. Here's the workflow for function tracing, call-graph timing, IRQ latency, and isolated trace instances.
Your shell scripts crash. trap EXIT is why the mess does not have to survive.
The shell trap builtin intercepts signals and the EXIT pseudo-signal to run cleanup code regardless of how a script dies. Most scripts skip it entirely. Here is how to use it correctly, including the subshell gotchas and the ERR trap edge cases that bite people.
The OOM killer was doing its job
A slow memory leak ran undetected for five weeks because the kernel's out-of-memory killer, executing its heuristic correctly, kept choosing the monitoring agent over the leaking service. The pager never fired. The monitoring gaps were there in the data the whole time.
ps lies about memory. /proc/smaps_rollup does not.
RSS from ps aux double-counts shared pages and makes every process look more expensive than it is. PSS from /proc/PID/smaps_rollup gives you actual per-process memory ownership. Here is how to read it.
ss -i shows you what netstat never could: TCP internals live
Everyone knows ss -tulnp. Almost nobody uses ss -i, which surfaces congestion window size, RTT, retransmit counts, and send/receive buffer fill — directly from the kernel, no tool required.
nohup, disown, setsid: what each actually does and which one you want
Three tools for keeping a process alive after you close the terminal. They do not do the same thing. Here is the kernel-level difference, and when each one applies.
ip netns is what container networking actually is. No daemon required.
Network namespaces give a process its own routing table, its own interfaces, and its own firewall rules — in milliseconds. Docker creates them. Kubernetes creates them. You can too, from ip, without touching a container runtime.
bpftrace gives Linux what FreeBSD had in 2005. The one-liners are worth the wait.
dtrace showed up on Solaris in 2004, shipped in FreeBSD 7 in 2008, and Linux users spent the next decade pretending strace was sufficient. bpftrace is the real answer — dynamic kernel tracing, histograms, stack walks, zero overhead when idle.
taskset pins a process to specific CPUs. numactl keeps its memory local too.
The Linux scheduler moves processes between cores. Sometimes that churn is the bottleneck — cold cache lines, NUMA cross-node fetches, latency spikes with no obvious cause. taskset and numactl are blunt instruments that work.
Your kernel ships a CPU profiler. perf(1) is the key.
perf stat hands you hardware performance counters — cycles, cache misses, branch mispredictions — in seconds. perf record samples call stacks at full speed. Both are already installed and beat any SaaS APM for understanding CPU-bound problems.
strace -c is a profiler. Stop spraying and praying.
Most people open strace, get buried in output, and close the terminal. Four flags fix that — -c counts, -e filters, -P follows a single path, -T times each call. Here is how strace is actually supposed to work.
Your container is six clone(2) flags. nsenter gets you back in.
Containers are six kernel namespaces and nothing else. nsenter gets you inside from the host without docker exec, without touching the image, and without whatever tooling the vendor decided to ship.
Git's SHA-256 transition is finally not vaporware
Six years after the object-format flag landed in 2.29, SHA-256 repos are actually getting real hosting support. The design they ended up with — capability advertisement plus translation proxies — is genuinely clever. I hate that it took this long and I respect how they did it.
Python finally killed the GIL. The code that needed it left years ago.
Free-threaded CPython is stable in 3.14. After thirty years, import threading will do what you thought it did. RodHat on why the right fix took this long, who actually benefits, and why the Python you probably care about is either unaffected or slower.
dig +trace, and how to tell whose DNS is actually lying to you
"It's a DNS problem" is where debugging stops. +trace walks the delegation from the root yourself, +norecurse asks a resolver what it has cached without letting it go fetch, and together they tell you whether the bad answer is the zone, the resolver, or the client.
pledge() turns 10. Linux still doesn't have anything half as clean.
OpenBSD's pledge(2) landed in 5.9 in 2016. Ten years later, Linux has Landlock (good) and seccomp-bpf (powerful and painful) and still nothing that lets a process sandbox itself in a single readable line. RodHat on why API simplicity is a security property.
Your cron job is running twice and that's why the numbers are wrong
A job that usually takes four minutes on a five-minute schedule will eventually take six, and then you have two copies racing. flock fixes it in one line — and the pidfile you were about to write instead is broken in ways flock isn't.
Unprivileged eBPF is getting locked out by default. It's five years overdue.
The Linux kernel project is landing a config change that makes kernel.unprivileged_bpf_disabled permanent by default. RodHat on why unprivileged BPF was always an attack surface in a trenchcoat, and why your bpftrace workflow is fine.
Observability priced per gigabyte means you log least when you need it most
Ingest-based pricing creates an incentive that fires exactly backwards: the incident that generates the most telemetry is the one that costs the most to observe. Teams respond by sampling, and then wonder why the postmortem has a gap in the middle.
The scary FreeBSD advisory this week isn't an RCE — it's a privilege check that slipped
FreeBSD-SA-26:53.ktrace is a regression in how ktrace is gated inside jails. No dramatic exploit, just an isolation boundary that quietly stopped meaning what you thought it meant.
The observability platform was down during the outage. Good. Now do it with your hands.
A junior engineer froze during a production fire because the SaaS dashboard that watches production was part of the fire. RodHat on the tools that were on every Unix box before the kid was born, the racket that sold competence back to us as a monthly invoice, and why the fire isn't out.
Let pf keep the ban list so you don't have to
pf tables and anchors turn a static ruleset into a firewall that bans its own attackers and updates without a reload. Real pfctl commands, real persistence, real expiry — not a cron job that greps auth.log like it's 2004.
Read a wedged process with your hands, not a dashboard
A process is hung and the graphs won't say why. Here's the /proc, ps, strace, and dmesg walk that finds a stuck worker in about ninety seconds — no agent, no vendor, no login.
Your restart policy is deleting the crime scene
Automatic restarts keep services available, but an aggressive restart loop can erase the timing, logs, cores, and state needed to understand why a process failed. Preserve evidence before recovery becomes amnesia.
The health check was green because it was checking the wrong damn thing
A service can answer HTTP 200 while its queue is wedged, its database writes are failing, and every useful request is dying. RodHat on health checks that prove process existence instead of service capability.
Your shell pipeline succeeded because the last command was polite
A failed producer can disappear behind a successful consumer. RodHat explains pipeline exit status, pipefail, PIPESTATUS, and how to stop backup scripts from lying.
"Cannot assign requested address" is not a DNS problem. You're out of ports.
A client making thousands of short-lived outbound connections exhausts the ephemeral range, and every tuple sits in TIME_WAIT for a minute afterwards. Here's how to confirm it in one command and the three fixes, ranked by how much you'll regret them.
The diff nobody wrote and everybody approved
AI coding tools now generate a meaningful percentage of commits at shops that have adopted them. RodHat on what happens to institutional knowledge when the author of the diff has never been paged.
We wrote the retry logic to survive a blip. It turned a blip into four hours.
One backend got slow for ninety seconds. Every client retried, in lockstep, three times, with no jitter. The retries were larger than the original traffic, the backend never recovered, and every fix we tried made it worse until we did the thing nobody wanted to do.
awk has hash maps, and that's why you don't need the Python script
Most people's awk stops at print $7. Associative arrays, END blocks and a two-line accumulator turn it into the right tool for ninety percent of the log questions people currently write a script for — and it runs on a box with nothing installed.
Another vendor decided the CLI was holding their users back. They're wrong.
A DevOps tooling vendor is sunsetting their CLI in favor of a "streamlined cloud dashboard experience." RodHat has opinions, and a survival guide.
zfs send is the backup tool you already have and probably aren't using
An incremental zfs send moves exactly the blocks that changed, verified end to end, with no file-tree walk at all. The full replication pipeline is one command, and the resume token turns a failed 3TB transfer from a restart into a continuation.
Debug a TLS handshake by hand with openssl s_client
"Certificate verify failed" is four different bugs wearing one error message. s_client shows you the chain the server actually sent, in order, with the verify result per link — which is how you find the missing intermediate in about twenty seconds.
The billing job ran twice for six weeks and the totals still balanced
No errors. No alerts. The reconciliation report matched every single day. And a few hundred customers were being double-charged, because the job that generated the charges and the job that verified them were the same code with the same bug.
Get a real disk latency histogram in one line of bpftrace
iostat gives you an average, and averages hide the tail that's actually hurting you. A four-line bpftrace program prints a log2 histogram of block I/O latency per device, live, on a production box, with no agent and no restart.
Five shell one-liners I actually use, not the ones that show up in every 'top 10' list
Real, still-in-daily-use shell one-liners for finding what's eating memory, what's holding a port, and what changed in a config file — no clickbait filler.
Process supervision is a 40-line problem. It has been solved since 1997.
You don't need a service manager with a DNS resolver in it to restart a daemon that died. daemontools, runit and s6 do supervision trees, log rotation and clean shutdown in a handful of files — and the design idea underneath them is worth stealing even if you never install one.
Certificate lifetimes are collapsing to 47 days and your renewal process is a person
The CA/Browser Forum voted to phase TLS certificate validity down to under seven weeks by 2029. If your renewal runbook has a human in it, the schedule just became your problem — and that's the actual point of the change.
find | xargs is a filename injection bug you've been shipping for years
Whitespace, newlines and quotes in filenames turn the standard find-pipe-xargs idiom into something that deletes the wrong files. -print0 and -0 fix it, find -exec + fixes it better, and there's one case where only one of them works.
Find out which syscall is actually eating your latency with one dtrace one-liner
A real dtrace/bpftrace one-liner for finding the syscall responsible for tail latency, plus why strace -T won't get you there.
The disk is full and du says it isn't. It's an open file descriptor.
df reports 100%, du walks the whole tree and finds nothing. The space is in a file somebody deleted while a process still had it open — the inode lives until the last fd closes. lsof +L1 finds it in one command.
Stop guessing what's listening on that port
netstat -an tells you a socket is open. It does not tell you which process opened it, which user owns it, or whether the accept queue is already overflowing. ss and sockstat do, and they take the same three seconds.