Split horizon DNS, and the day my own domain broke a cron job

I run split horizon DNS on a domain I own. Inside the network, names resolve to the internal reverse proxy. Outside, to the public edge. It broke twice: a cron job could not reach my own public site from inside, and my laptop’s VPN resolver depended on the VPN. This covers both, and the fix that did not hold.

Why split it at all

Without it, internal clients get the public answer. Traffic goes out to the internet edge and back in again to reach a machine on the same switch.

Doing it on a real domain rather than an invented one is worth the effort. .local is already taken: RFC 6762 says any query for a name ending in .local must go to multicast DNS. A made-up TLD gets you no public certificate, so you end up running a private CA and installing root certs on a television.

With a real domain, internal names get normal HTTPS. I use one wildcard certificate issued over DNS-01, not one per host. Let’s Encrypt submits every certificate it issues to Certificate Transparency logs, so per host certificates publish every internal hostname. A wildcard publishes the wildcard. DNS-01 is also the only Let’s Encrypt challenge that can issue one.

A wildcard matches exactly one label. *.example.com covers app.example.com, not a.b.example.com and not example.com itself. Names two labels deep need a second wildcard.

Why my cron job got the wrong answer

A scheduled job on the Pi fetches a URL on the public site. It worked for a while, then stopped, silently, the way scheduled jobs do. The chain, once unpicked:

• internal DNS answered the apex of the domain with the internal reverse proxy, which is the whole point of split horizon
• the internal answer was AAAA only
• the client connected over IPv6 and sent the public hostname in SNI
• the proxy had no certificate for that name, so the handshake failed

Every layer behaved exactly as configured. The job wanted the public site and the network kept giving it the private one.

The fix was two small changes for that one job. Look the name up through a public resolver, then pin that answer with --resolve:

ip=$(dig +short @1.1.1.1 example.com A | head -n1)
curl --resolve "example.com:443:$ip" https://example.com/wp-cron.php

--resolve gives curl an address for one host and port pair. The curl docs call it an /etc/hosts alternative on the command line, and it applies even if --ipv4 or --ipv6 is set. Nothing else on the machine is affected.

The general shape is worth keeping: split horizon breaks anything that needs the public version of a name from inside. Health checks, cron jobs, webhooks that loop back, certificate validation. All of them get the internal answer.

Why did my VPN lose internal DNS?

The Mac would stop resolving internal names for about five minutes at a time, several times an hour. A DNS cache flush “fixed” it.

The VPN is NetBird. It pushes a match domain, so macOS sends every query for my domain to NetBird’s own resolver. NetBird forwarded those to a nameserver group with exactly one upstream, the internal DNS server. And the Mac routed the network that server sits on through the tunnel.

So when the tunnel flapped, the upstream became unreachable and NetBird stopped answering for the domain. Queries fell through to public DNS, and internal names came back with their public records. The client’s own event log showed it the whole time, alternating Nameserver group unreachable and recovered.

The fix I made was a second upstream on a different path: the resolver on the Pi, reached over the local network instead of the tunnel.

One nuance cost extra time. A secondary DNS server handed out by DHCP does not fix this on the Mac. NetBird’s match domain nameservers become scoped resolvers, visible in scutil --dns, and those answer for the domain regardless of what DHCP says. The DHCP change is a separate fix, for every other device on the network.

Did the second upstream fix it?

No. The fix was decorative. I added it on 17 August, and on 7 September the outages were back with both upstreams failing at once.

The resolver on the Pi was offline, so the second path led nowhere. The internal DNS server had been healthy the whole time. My laptop’s packets to it were being dropped by the gateway’s intrusion prevention, which had banned my own VPN address. The tell was that DNS and SSH to the gateway died while the websites behind it kept working.

That is fixed now, and it needs a post of its own. The earlier version of this post called the second upstream the fix. It was the right idea, and it was not the problem.