The internal API migration had gone well. Two weeks of planning, a test cutover on staging, a maintenance window on a Tuesday night. We updated the DNS record, pushed the new servers live, and confirmed with dig that the A record was pointing at the right addresses.
Then we turned off the old servers and the application fell over.
The first thing you do when a service can’t reach a hostname is check DNS. That is the correct instinct. And DNS was correct.
$ dig +short api.internal
10.20.30.40That was the new server. The new server was up. The new server was accepting connections on port 443. We could curl it directly by IP. We could curl it from the application host by IP. We could not reach it by name, from the application.
The service logs said “connection refused.” The new server’s logs showed no incoming connections. The old server, which was supposed to be off, was also logging nothing. Nobody was connecting to anybody.
The next step was strace.
$ strace -f -e trace=network -p <service pid>
...
connect(12, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("10.10.10.10")}, 16) = -1 ECONNREFUSED10.10.10.10.
That was the old server. The one we had just turned off. The application was trying to connect to a server that did not exist anymore, and the reason dig had shown us the right answer is that dig is a DNS client and the application is not using DNS.
Here is the thing that trips people up.
On Linux, name resolution is not “ask DNS.” Name resolution is whatever nsswitch.conf says it is. Open /etc/nsswitch.conf on any production box and find the hosts: line:
hosts: files dnsThat line says: to resolve a hostname, first check /etc/hosts (that’s files), and if that returns an answer, stop. Only go to DNS if /etc/hosts has nothing.
The entry in /etc/hosts:
$ grep api.internal /etc/hosts
10.10.10.10 api.internalThere it was. Added three years ago when we were standing up the internal API, before the DNS zone was fully delegated. Someone had dropped it in /etc/hosts to unblock their local testing and it had stayed there through three OS reinstalls and two infrastructure migrations, because /etc/hosts does not self-clean and nobody looks at it unless something is wrong.
dig bypasses NSS entirely. It constructs a DNS query directly against the servers in /etc/resolv.conf. It does not check /etc/hosts, it does not care about nsswitch.conf, it does not go through getaddrinfo(3). It talks to DNS and reports what DNS says. The DNS was correct. The application uses getaddrinfo(), which respects nsswitch.conf, which checked /etc/hosts first and found a stale answer and stopped.
Two stacks. One correct, one wrong. dig tested the correct one and reported success, and the application was living in the wrong one the whole time.
The correct tool for debugging application name resolution is getent:
$ getent hosts api.internal
10.10.10.10 api.internalgetent goes through NSS. It calls the same getaddrinfo() path your application does. If getent returns the wrong answer, your application will get the wrong answer. If dig returns the right answer but getent returns the wrong answer, you have found the problem: something between DNS and your application is intercepting or overriding.
We removed the /etc/hosts entry:
$ getent hosts api.internal
10.20.30.40 api.internalService came back. Total downtime was about twenty-five minutes, which was twenty-five minutes longer than it should have been given that the answer was in a text file we could have checked in thirty seconds.
/etc/hosts is the obvious one but it is not the only layer. On modern Ubuntu and Debian boxes with systemd-resolved, the NSS configuration often looks like this:
hosts: files mdns4_minimal [NOTFOUND=return] dns resolve [!UNAVAIL=return] mdns4That is substantially more complicated than files dns. The mdns4_minimal plugin handles .local hostnames via multicast DNS. The [NOTFOUND=return] directive says that if mdns4_minimal explicitly says the name does not exist, stop looking. The resolve plugin queries systemd-resolved, which has its own cache with its own TTLs that may or may not agree with what the actual DNS says.
systemd-resolved maintains a cache. The cache respects TTLs, mostly. But it also has a negative cache: if it tried to resolve a name and got NXDOMAIN, it remembers that and returns NXDOMAIN for subsequent queries until the TTL on the negative response expires. If your DNS was temporarily misconfigured and resolved got a NXDOMAIN, your box may refuse to resolve that name for the next twenty minutes or two hours even after you fix the DNS. resolvectl flush-caches clears it.
There is also nscd, the Name Service Caching Daemon, which some older distributions run. nscd caches NSS results at a different layer, with different TTLs, and it is controlled by /etc/nscd.conf. Most modern distributions do not run nscd by default, but if you are debugging name resolution and nothing else explains the problem, check whether nscd is running.
The thing about the dig trace DNS debugging guide is that it tests the DNS layer. It is the right tool for verifying that the DNS record is correct. It is the wrong tool for verifying that the record your application will use is correct. Those are different questions and they get different answers from different tools.
After we fixed the outage, we went looking for other stale entries in /etc/hosts. There were eleven.
Four of them pointed at servers that had been decommissioned. Two pointed at services that had moved to different IPs. Three pointed at development boxes that no longer existed. One was a personal laptop that had been gone for two years. One was correct, put there on purpose by a colleague who wanted to route a specific hostname through a different IP for local development, which is fine, but nobody else knew it was there and the comment explaining it had been deleted at some point.
We turned the file into configuration management, added a lint step that checked every entry against current DNS, and ran the linter weekly. The number of stale entries in the next audit was zero.
None of this required a particularly sophisticated tool. getent hosts is in every Linux distribution, has been for thirty years, and costs nothing. The gap was that everyone on the team knew to reach for dig when DNS was the question, and nobody had internalized that DNS is not name resolution.
The pattern fits alongside every other silent override on the networking and DNS shelf: name resolution is layered, each layer can contradict the one below it, and the question “what IP does this hostname resolve to” is ambiguous until you specify which stack you are asking. dig answers it for DNS. getent answers it for the stack your application uses. Those are not the same question, and on a busy production box with three years of accumulated /etc/hosts entries and a systemd-resolved cache that has seen better days, they are not going to return the same answer.
Use getent. Check /etc/hosts. Check nsswitch.conf. Then use dig to verify the DNS itself is right. That order, not the other way around.