Split horizon DNS, and the day my own domain broke a cron job

Split horizon DNS means one name answering differently depending on who asks. Inside the network, a hostname resolves to a private address and traffic never leaves the switch. Outside, the same name resolves to a public address. It is a genuinely good pattern, and it is also the pattern that broke a scheduled job in a way that took an embarrassing amount of time to see.

Why split it at all

The alternative to split horizon is hairpinning: internal clients resolve the public address, leave through the router, and come back in through the same router to reach a machine two metres away. Some routers handle that. Many handle it badly. All of them add latency for no reason and put internal traffic through the WAN interface. 🪃

Doing it on a domain you actually own, rather than on .local or an invented TLD, is the right call and worth defending. Real domain, real certificates. Internal names get proper HTTPS with no warnings and no private certificate authority to distribute to every device. If you make up a TLD you cannot get a public certificate for it, and then you are the person maintaining a CA and installing root certs on a television.

The certificate approach that goes with it: one wildcard, issued over DNS-01, rather than a certificate per host. Not for convenience. Every publicly trusted certificate is published to Certificate Transparency logs, and those logs are indexed and searchable. Per host certificates publish an internal hostname to a public index every time one is issued. A wildcard publishes the wildcard, once.

The failure

A scheduled job on the always on Pi had to fetch a URL on the public site. It had worked for a while. Then it stopped, silently, the way scheduled jobs do. ⏰

The chain, once unpicked:

• internal DNS answers the apex of the domain with the internal reverse proxy, which is the whole point of split horizon
• that internal record had only AAAA answers on the internal resolver
• the client preferred the v6 answer, connected, and presented the public hostname in SNI
• the internal proxy had no certificate matching that name on that path, so the handshake failed

Every layer behaved exactly as configured. The job just wanted the public site and the network kept helpfully giving it the private one.

Two fixes, both small. Resolve through a public resolver for that specific job, and pin the address so the answer cannot drift:

curl --resolve 'example.com:443:203.0.113.10' https://example.com/wp-cron.php

--resolve is the useful half. It overrides resolution for one hostname and port on one command, without touching /etc/hosts and without affecting anything else on the machine. For a cron job that must reach the public edge of a domain you also serve internally, it is the correct tool.

The general shape of the bug is worth keeping: split horizon breaks anything that needs to reach the public version of a name from inside. Health checks, cron jobs, webhooks that loop back, certificate validation. All of them will get the internal answer, and all of them will be confusing about it 🙂

The other DNS outage, which was circular

Before that one, a different failure: the Mac would randomly stop resolving internal names for about five minutes at a time, several times an hour, and a DNS cache flush appeared to fix it. That “appeared” was doing a lot of work.

The mesh VPN pushes a match domain resolver, so the operating system sends every query for that domain to the overlay’s own resolver rather than to DHCP’s. The overlay forwarded those to a nameserver group that had exactly one upstream: the internal DNS server.

And the Mac routes the network that server sits on through the tunnel.

So: tunnel flaps, the upstream becomes unreachable, the overlay marks the nameserver group down and stops answering for the domain, queries escape to public DNS, and the internal name resolves to whatever the public record says. The evidence was sitting in the client’s own event log the whole time, alternating unreachable and recovered, with the single upstream named in the metadata.

A resolver group whose only upstream is reachable only through the tunnel is a resolver group that fails exactly when it is needed. The fix was a second upstream on a different path: the resolver on the Pi, which sits on a different network and is reached over the local interface rather than the tunnel. Reachable precisely when the tunnel is the broken thing.

One nuance cost extra time. Adding a secondary DNS server in DHCP does not fix this. macOS’s per domain resolver from the VPN takes precedence over anything DHCP hands out for that domain. The DHCP change is a different fix, for all the other devices, for a different failure. Conflating them means changing something that cannot possibly help and then believing you have fixed it 🙃

What to take from both

Two outages, one shape. A resolver that depends on the thing it is used to reach.

The check is short. For every name your infrastructure needs, ask what has to be up for that name to resolve, then ask whether that thing is inside or outside the failure you are trying to survive. If the answer is inside, it is not redundancy, it is a longer chain. ⛓️

That is the end of this series for now. It starts with the hardware, if you came in at the wrong end.