RodHat's Tips From the Console
Next-level shit. Not "here's how to ls -la."
strace -c is a profiler. Stop spraying and praying.
Most people open strace, get buried in output, and close the terminal. Four flags fix that — -c counts, -e filters, -P follows a single path, -T times each call. Here is how strace is actually supposed to work.
Your container is six clone(2) flags. nsenter gets you back in.
Containers are six kernel namespaces and nothing else. nsenter gets you inside from the host without docker exec, without touching the image, and without whatever tooling the vendor decided to ship.
socat is the Swiss Army knife you keep reaching past
netcat forwards a port. socat forwards a port, wraps it in TLS, splices it to a Unix socket, proxies serial-over-TCP, and lets you inject raw bytes mid-stream. You already have it installed. Here is how to actually use it.
dig +trace, and how to tell whose DNS is actually lying to you
"It's a DNS problem" is where debugging stops. +trace walks the delegation from the root yourself, +norecurse asks a resolver what it has cached without letting it go fetch, and together they tell you whether the bad answer is the zone, the resolver, or the client.
Your cron job is running twice and that's why the numbers are wrong
A job that usually takes four minutes on a five-minute schedule will eventually take six, and then you have two copies racing. flock fixes it in one line — and the pidfile you were about to write instead is broken in ways flock isn't.
Two uplinks, one box, and the reply going out the wrong interface
The routing table picks a route by destination. When you have two upstreams, replies to traffic that arrived on the second one leave via the first, get dropped by the upstream's anti-spoofing, and vanish. ip rule and a second routing table fix it properly.
Upgrade like you can undo it, because with bectl you can
A ZFS boot environment is a bootable clone of your root dataset. Make one before every upgrade and a wrecked kernel becomes a reboot, not a recovery-media evening. The bectl walk, and the loader trick that saves you when the new one won't boot.
Let pf keep the ban list so you don't have to
pf tables and anchors turn a static ruleset into a firewall that bans its own attackers and updates without a reload. Real pfctl commands, real persistence, real expiry — not a cron job that greps auth.log like it's 2004.
Read a wedged process with your hands, not a dashboard
A process is hung and the graphs won't say why. Here's the /proc, ps, strace, and dmesg walk that finds a stuck worker in about ninety seconds — no agent, no vendor, no login.
Your restart policy is deleting the crime scene
Automatic restarts keep services available, but an aggressive restart loop can erase the timing, logs, cores, and state needed to understand why a process failed. Preserve evidence before recovery becomes amnesia.
Stop waiting beside tcpdump like it owes you an incident
Intermittent network failures do not happen while you are watching. Use tcpdump's rotating capture files, size limits, and post-trigger preservation to keep the packets that existed before the pager fired.
Your shell pipeline succeeded because the last command was polite
A failed producer can disappear behind a successful consumer. RodHat explains pipeline exit status, pipefail, PIPESTATUS, and how to stop backup scripts from lying.
"Cannot assign requested address" is not a DNS problem. You're out of ports.
A client making thousands of short-lived outbound connections exhausts the ephemeral range, and every tuple sits in TIME_WAIT for a minute afterwards. Here's how to confirm it in one command and the three fixes, ranked by how much you'll regret them.
awk has hash maps, and that's why you don't need the Python script
Most people's awk stops at print $7. Associative arrays, END blocks and a two-line accumulator turn it into the right tool for ninety percent of the log questions people currently write a script for — and it runs on a box with nothing installed.
zfs send is the backup tool you already have and probably aren't using
An incremental zfs send moves exactly the blocks that changed, verified end to end, with no file-tree walk at all. The full replication pipeline is one command, and the resume token turns a failed 3TB transfer from a restart into a continuation.
The six rsync flags worth knowing, and the one that builds you free snapshots
-a doesn't preserve what you think it preserves. --inplace and --partial solve opposite problems and combining them wrong loses data. And --link-dest turns rsync into a deduplicating snapshot system in one flag.
Debug a TLS handshake by hand with openssl s_client
"Certificate verify failed" is four different bugs wearing one error message. s_client shows you the chain the server actually sent, in order, with the verify result per link — which is how you find the missing intermediate in about twenty seconds.
Stop your backup job from taking production down with it
nice does nothing for I/O. ionice only works on one scheduler that most systems no longer use. cgroup v2 io.latency and io.max are the two knobs that actually work, and they work on the thing you already have.
Get a real disk latency histogram in one line of bpftrace
iostat gives you an average, and averages hide the tail that's actually hurting you. A four-line bpftrace program prints a log2 histogram of block I/O latency per device, live, on a production box, with no agent and no restart.
Five shell one-liners I actually use, not the ones that show up in every 'top 10' list
Real, still-in-daily-use shell one-liners for finding what's eating memory, what's holding a port, and what changed in a config file — no clickbait filler.
Process supervision is a 40-line problem. It has been solved since 1997.
You don't need a service manager with a DNS resolver in it to restart a daemon that died. daemontools, runit and s6 do supervision trees, log rotation and clean shutdown in a handful of files — and the design idea underneath them is worth stealing even if you never install one.
find | xargs is a filename injection bug you've been shipping for years
Whitespace, newlines and quotes in filenames turn the standard find-pipe-xargs idiom into something that deletes the wrong files. -print0 and -0 fix it, find -exec + fixes it better, and there's one case where only one of them works.
ZFS boot environments + jails will do 90% of what you're using Docker for, and the other 10% is the part you don't actually need
A real walkthrough of FreeBSD jails on ZFS datasets as a lighter, more honest isolation model than a container runtime for a lot of workloads.
Your ssh is reconnecting every single time. It doesn't have to.
ControlMaster reuses one TCP connection and one authentication for every subsequent session to the same host. Combine it with ProxyJump and a Match block and the difference between a 900ms hop and a 20ms one is four lines of config you write once.
No, ZFS did not eat your RAM. Read arc_summary before you tune anything.
The ARC is designed to consume all the memory you aren't using and give it back under pressure. But it does give it back slowly, and there are two real cases where you should cap it. Here's how to tell which situation you're actually in.
Find out which syscall is actually eating your latency with one dtrace one-liner
A real dtrace/bpftrace one-liner for finding the syscall responsible for tail latency, plus why strace -T won't get you there.
The disk is full and du says it isn't. It's an open file descriptor.
df reports 100%, du walks the whole tree and finds nothing. The space is in a file somebody deleted while a process still had it open — the inode lives until the last fd closes. lsof +L1 finds it in one command.
Stop guessing what's listening on that port
netstat -an tells you a socket is open. It does not tell you which process opened it, which user owns it, or whether the accept queue is already overflowing. ss and sockstat do, and they take the same three seconds.