DNS: The Gaslighting Layer
It’s Probably Not the Network Series: Issue 2 of 5
There are few phrases in IT more dangerous than:
“The network is down.”
Not because the network is innocent.
Let’s not get carried away.
The network has committed plenty of crimes. It has dropped packets, formed routing loops, silently blackholed traffic, let spanning tree have feelings, and once made someone troubleshoot a “bad switch” that turned out to be a chair leg pinching a patch cable.
But sometimes, the network gets framed.
And no layer frames the network better than DNS.
DNS is the gaslighting layer of infrastructure. It can make a perfectly reachable service look dead, make one user swear the app is broken while another user says “works for me,” and make an entire troubleshooting bridge spiral into philosophical debate because a hostname points to something that was decommissioned during the Obama administration.
The best part?
DNS is usually doing exactly what someone configured it to do.
Three years ago.
Temporarily.
During a migration.
That nobody documented.
Welcome to Issue 2 of It’s Probably Not the Network, where we stop blaming switches for name resolution crimes.
The Classic DNS Incident
Here’s the usual scene.
A user reports:
“The app is down.”
The app team checks the service and says:
“Everything is healthy.”
The server team checks CPU, memory, and disk and says:
“Looks fine.”
The firewall team checks the rule and says:
“Traffic is allowed.”
The network team pings the IP and says:
“Network is fine.”
Then someone finally runs:
nslookup app.company.com
And suddenly the room gets quiet.
Because app.company.com resolves to an IP address that no longer exists.
Or it resolves to the public IP from inside the network.
Or VPN users get a different answer than LAN users.
Or the CNAME chain points to a vendor hostname that changed last week.
Or the client has cached an old response and is confidently walking into a wall.
DNS doesn’t always fail like a service outage.
Sometimes it fails like a liar with perfect uptime.
Why DNS Makes Everything Look Broken
DNS issues rarely announce themselves as DNS issues.
Nobody opens a ticket saying:
“Hello, I believe my recursive resolver path may be returning an unexpected record due to split-horizon DNS misalignment.”
No.
They say:
“The internet is broken.”
Or:
“The app won’t load.”
Or:
“VPN is broken.”
Or:
“The firewall must be blocking it.”
Or the truly cursed classic:
“It worked yesterday.”
DNS problems often look like:
A website will not load
An app works by IP but not by hostname
Internal users and external users get different results
VPN users resolve the wrong address
One subnet works while another fails
A new DNS record “hasn’t propagated”
Authentication fails because reverse DNS is wrong
Monitoring alerts fire against the wrong host
Certificates don’t match the hostname being used
Half the environment works because half the environment cached the old answer
That’s what makes DNS so dangerous.
A routing problem usually looks like routing.
A firewall problem usually leaves logs, sessions, drops, or at least enough evidence for someone to argue about over coffee.
DNS, on the other hand, quietly hands the client the wrong address and walks away like it did its job.
Because technically?
It did.
Step 1: Stop Troubleshooting “The App”
The first mistake in DNS troubleshooting is accepting vague nouns.
“The app is down.”
“The site is broken.”
“The server is unreachable.”
“The thing doesn’t work.”
That is not enough.
You need the actual name being accessed.
Not the friendly name.
Not the bookmark description.
Not “the portal.”
The actual fully qualified domain name.
Examples:
app.company.com
api.company.com
login.company.com
files.internal.company.com
prod-lb-east.vendorcloud.net
This matters because “the app” may involve:
A web frontend
An API endpoint
A login provider
A database listener
A load balancer
A CDN hostname
A vendor-managed CNAME
A redirect chain
A certificate name
An internal-only DNS zone
A public DNS zone
A search suffix doing something deeply unholy
You are not troubleshooting “the app.”
You are troubleshooting a name.
And DNS only cares about names.
Not vibes.
Step 2: Test Name vs. IP
Before blaming the network, test whether the problem is name resolution or actual reachability.
Basic test:
ping app.company.com
ping 10.20.30.40
Better test for web apps:
curl -v https://app.company.com
curl -v https://10.20.30.40
On Windows PowerShell:
Test-NetConnection app.company.com -Port 443
Test-NetConnection 10.20.30.40 -Port 443
What you are trying to determine:
Does the hostname fail, but the IP works?
If yes, DNS is now a suspect.
Not convicted.
Not thrown into the root cause report yet.
But definitely sitting in the interview room with stale records in its pockets.
If both hostname and IP fail, you may have a routing, firewall, server, application, or service issue.
If IP works but hostname fails, DNS just moved way up the suspect list.
Step 3: Ask the Client What It Thinks
The client’s DNS answer is what matters.
Not what the DNS console shows.
Not what the admin remembers configuring.
Not what someone put in the migration spreadsheet.
The client is going to connect to whatever IP address it resolves.
So ask the client.
On Windows:
nslookup app.company.com
Resolve-DnsName app.company.com
On Linux or macOS:
dig app.company.com
nslookup app.company.com
Look for a few key things:
What IP address came back?
Which DNS server answered?
Was the response A, AAAA, CNAME, or something else?
Is the answer internal or external?
Is it the expected IP?
Is there more than one answer?
That “which DNS server answered” part is important.
Because two clients can query the same hostname and receive different answers if they are using different resolvers.
That is not necessarily broken.
That may be split DNS.
Or load balancing.
Or geo DNS.
Or a misconfigured VPN client.
Or one DNS server living its best undocumented life.
Step 4: Ask a Specific DNS Server
This is where DNS troubleshooting starts getting useful.
Do not just ask “DNS.”
Ask a specific resolver.
For example:
nslookup app.company.com 10.10.10.10
nslookup app.company.com 8.8.8.8
Or with dig:
dig @10.10.10.10 app.company.com
dig @8.8.8.8 app.company.com
Now you can compare answers.
Example:
Internal DNS:
app.company.com -> 10.20.30.40
Public DNS:
app.company.com -> 52.10.20.30
VPN client:
app.company.com -> 52.10.20.30
That tells a story.
Internal LAN users get the private IP.
External users get the public IP.
VPN users are also getting the public IP, even though they should probably get the private IP.
That might point to:
VPN DNS settings
Split-tunnel DNS behavior
Wrong resolver order
Missing internal DNS suffix
Client DNS cache
Security agent interference
A VPN profile built by someone who was clearly under duress
When two DNS servers give two different answers, congratulations.
You have entered the DNS multiverse.
Split-Horizon DNS: Same Name, Different Reality
Split-horizon DNS is when the same hostname returns different answers depending on where the query comes from.
Example:
Inside the network:
app.company.com -> 10.20.30.40
Outside the network:
app.company.com -> 52.10.20.30
This is common and often completely valid.
Internal users should hit the private load balancer.
External users should hit the public load balancer.
VPN users might need the internal answer.
Guest Wi-Fi might need the external answer.
Cloud workloads might need something else entirely because apparently every environment needs its own emotional support DNS design.
Split DNS is not bad.
Undocumented split DNS is bad.
Split DNS managed by six teams, two vendors, three appliances, and one retired engineer’s ancient notes folder is how outages become lore.
When split DNS is involved, always ask:
Where is the client located?
What DNS server is the client using?
What answer should that client receive?
What answer did it actually receive?
DNS is all about perspective.
Which is a polite way of saying:
Same name.
Different reality.
Enjoy.
CNAME Chains: The DNS Treasure Hunt Nobody Asked For
A DNS name may not point directly to an IP address.
It may point to another name.
Which points to another name.
Which points to another name.
Which finally points to an address managed by a vendor with a status page that says “all systems operational” during an active meteor strike.
Example:
app.company.com
CNAME app-prod.company.net
CNAME prod-east-lb.vendorcloud.com
A 34.123.45.67
CNAMEs are useful.
CNAME chains are where the fun begins.
Possible problems:
One CNAME target no longer exists
The final A record changed
Internal DNS overrides only part of the chain
The certificate matches one hostname but not another
A vendor changed their DNS target
TTLs are inconsistent across the chain
Someone deleted an “unused” record that was load-bearing
That last one deserves respect.
In IT, “unused” often means:
“I personally do not understand what this does.”
Which is how production gets sacrificed to the cleanup gods.
To inspect a CNAME chain:
dig app.company.com
dig +short app.company.com
On Windows:
Resolve-DnsName app.company.com
You are looking for the full path from the name users type to the final address the client connects to.
Because the problem might not be the first record.
It might be the third alias in the chain wearing a trench coat.
TTL: The Lie Everyone Believes
TTL stands for Time To Live.
It tells DNS resolvers how long they can cache an answer.
Simple enough, right?
Adorable.
The problem is that people often think TTL means:
“I changed the DNS record, so everything should work now.”
No.
TTL means:
“Some systems may continue using the old answer until their cache expires, unless they ignore you, cache weirdly, or are haunted.”
DNS answers can be cached in many places:
Client operating system
Browser
Local resolver
Domain controller
ISP resolver
Public resolver
VPN client
Security agent
Load balancer
Proxy
Container runtime
Application process
That one appliance nobody logs into because the password is allegedly “somewhere”
On Windows, you can inspect and flush the DNS cache:
ipconfig /displaydns
ipconfig /flushdns
On Linux with systemd-resolved:
resolvectl statistics
sudo resolvectl flush-caches
On macOS:
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder
But flushing the local cache only helps the local cache.
It does not magically flush every resolver between the client and the authoritative DNS server.
For planned migrations, lower the TTL ahead of time.
Not during the cutover.
Not after everyone is angry.
Ahead of time.
If a record has a TTL of 24 hours and you change it five minutes before a migration, you did not perform a DNS cutover.
You performed a trust exercise.
Reverse DNS: The Thing Nobody Checks Until It Hurts
Forward DNS answers this:
What IP address belongs to this name?
Reverse DNS answers this:
What name belongs to this IP address?
Reverse lookups use PTR records.
Example:
dig -x 10.20.30.40
Or:
nslookup 10.20.30.40
Reverse DNS may matter for:
Logging
Monitoring
Email systems
Authentication workflows
Security tools
Legacy applications
Some certificate or trust checks
Kerberos-adjacent weirdness
Systems that get emotionally attached to hostnames
A reverse DNS issue may not stop every connection.
But it can absolutely break specific systems in deeply annoying ways.
Especially older systems.
And by “older systems,” I mean the ones labeled “temporary” in 2016 that now run payroll, badge access, and a business-critical Excel macro named final_final_v7_REAL.xlsm.
Forward DNS tells the client where to go.
Reverse DNS helps systems agree on what they are talking to.
And sometimes everyone disagrees with confidence.
The DNS Sanity Check
Before blaming the firewall, switch, router, ISP, cloud provider, VPN concentrator, or “the network,” run through this:
1. What exact FQDN is failing?
2. What IP should it resolve to?
3. What IP does the affected client actually get?
4. Which DNS server answered?
5. Does a different DNS server give a different answer?
6. Do internal and external resolvers agree?
7. Is split-horizon DNS expected?
8. Is there a CNAME chain?
9. Does every CNAME target resolve?
10. What is the TTL?
11. Could the client or resolver have a stale cache?
12. Does reverse DNS matter for this workflow?
13. Does the app work by IP but fail by hostname?
That checklist alone can save an hour of random pinging, emotional escalation, and someone saying “must be DNS” with no evidence.
Because “DNS issue” is not a root cause.
It is a weather report.
The real question is:
Where in the resolver path did the answer go wrong?
The Blame Funnel: DNS Edition
When DNS is suspected, do not jump straight to “DNS is broken.”
Walk it down.
User says app is down
↓
Identify the exact hostname
↓
Compare hostname test vs IP test
↓
Check client DNS answer
↓
Check which resolver answered
↓
Query known internal resolver
↓
Query known external resolver
↓
Inspect CNAME chain
↓
Check TTL and cache
↓
Check split DNS expectations
↓
Document evidence
↓
Escalate to the correct owner
The goal is not to win the argument.
Okay, fine.
The goal is mostly not to win the argument.
The real goal is to prove where the failure is happening so the right team can fix the right thing.
Preferably before the outage bridge becomes a support group.
Related Config Chaos
If this issue made you side-eye your DNS records, good.
That means the healing has started.
Here are a few related The Config Report issues that pair nicely with today’s DNS-induced identity crisis:
The Blame Funnel: How to Troubleshoot Without Randomly Pinging Things
The first issue in this series, where we talked about building a repeatable troubleshooting process before everyone starts firing ICMP into the darkness like it owes them money.Packet Crimes: Hunting Down the Jerk Flooding Your Network
For those moments when DNS is innocent, but something on the wire is absolutely acting suspicious.Configuration Drift Is Killing Your Network
Because sometimes the DNS record is wrong for the same reason everything else is wrong: nobody noticed reality and documentation broke up years ago.Why Most IaC Projects Fail, Even With the Right Tools
A nice companion for the “temporary DNS change became permanent infrastructure” crowd.CI/CD for Infrastructure: Stop Manually Yeeting Configs Into Production
For when DNS, firewall rules, load balancers, and configs need fewer heroic manual changes and more adult supervision.
Infrastructure rarely breaks in isolation.
It breaks in layers.
Like an onion.
Or a lasagna.
Or a change request approved at 4:57 PM on a Friday.
Keep reading with a 7-day free trial
Subscribe to The Config Report to keep reading this post and get 7 days of free access to the full post archives.