Tailnet-only routes returned 403 because Tailscale rewrote the source address

Traefik's IP allowlist for the tailnet range rejected tailnet clients because Tailscale's exit-node masquerade changed their source address, fixed with a mangle rule on all five nodes.

Context

My servers are joined into one Tailscale network. Several internal services run behind Traefik in Docker, and I keep them private by giving their routes an ipAllowList middleware that only accepts the tailnet range, 100.64.0.0/10 (the CGNAT block Tailscale uses). From outside the tailnet those routes are not reachable at all.

All five nodes also advertise themselves as exit nodes (0.0.0.0/0), because I use some of them as exit nodes from other devices. That means source NAT is switched on for forwarded traffic.

What I saw

Routes with the allowlist returned 403 to clients that were definitely on the tailnet. Traefik's access log showed the cause plainly: the ClientHost was the Docker bridge gateway address (a private 172.x address), not a tailnet address. Traefik was working correctly. From its point of view the request came from the Docker gateway, which is not in the allowed range.

Public routes were unaffected, and they still saw real client addresses.

Finding the cause

The clue was that the source address had been replaced somewhere between the network interface and the container. Docker publishes a port with a DNAT rule, so a packet from the tailnet arrives on tailscale0, is rewritten to the container's address, and is therefore forwarded, not delivered locally.

Reading Tailscale's firewall chains explained the rest:

• The ts-forward chain puts a mark (0x40000) on forwarded packets arriving from tailscale0.
• The ts-postrouting chain in the nat table masquerades every packet carrying that mark. That is how exit-node traffic gets out to the internet.
• A DNAT'ed packet headed for a container is also forwarded, so it picks up the same mark and is masqueraded on the way into the bridge. The container then sees the gateway address.

Why did some nodes not show the problem? The result depends on the order of rules in the FORWARD chain, and that order depends on which of tailscaled and dockerd started last, because whichever starts last inserts its rules first. Two nodes were broken. The other three were working only by luck, and a Tailscale auto-update would eventually have broken them as well.

I considered turning off exit nodes, but I use them, so that was not an option. I also looked at widening the allowlist to include the Docker bridge range. I rejected that because it would let any traffic that reaches the bridge, including masqueraded exit-node traffic, pass the check, and it defeats the point of the allowlist.

A debugging trap I noted: Traefik's access log does not record requests for hosts that have no route, so probing with a made-up hostname shows nothing. I measured with the packet counters of the masquerade chain (iptables -t nat -L ts-postrouting -v) and with a temporary probe route.

Root cause

Tailscale's masquerade rule matched tailnet traffic that was only passing through to a local container, not just traffic leaving through an exit node. Which rule won depended on startup order, so the same configuration gave different results on different nodes.

The fix

I clear the Tailscale mark on packets leaving into Docker networks, in the mangle table, which runs before nat:

iptables -t mangle -A POSTROUTING -o br-+     -m mark --mark 0x40000/0x40000 -j MARK --set-xmark 0x0/0x40000
iptables -t mangle -A POSTROUTING -o docker0  -m mark --mark 0x40000/0x40000 -j MARK --set-xmark 0x0/0x40000

Because nothing else manages that chain, rule order no longer matters. Exit-node traffic going out of the public interface keeps its mark and is still masqueraded.

Verification: on the two affected nodes the masquerade counter rose by 1 to 2 packets before the change and stayed at 0 after. A probe route with an allowlist went from 403 to 200.

What stops it happening again

The rules are persisted so they survive reboot: appended to the existing firewall script on four nodes, and on the fifth (a different setup) as a small dedicated systemd unit. Removing the change is the same rule with -D.

There is also a regression check, though it was not built for this. The monitoring node probes each tailnet-only app every 60 seconds over the tailnet, using a probe profile that does not accept 403 as healthy (the comment in the config says a 403 there means Traefik refused the source). A failing probe raises an EndpointDown alert after two minutes. If the masquerade came back, those apps would start returning 403 and page me. The exception is the few apps marked as shown but never alerted. I added these probes on 24 October, after the incident.

What I learned

A 403 from an allowlist is a statement about the address the proxy saw, so the first thing to check is the access log's client address, not the middleware config.

Behaviour that depends on startup order is a latent fault even when it works. Three of five nodes were fine only by luck, which made this a fleet-wide fix rather than a two-node patch.

Fix it at a layer that is under your control and ordered by design (here, the mangle table) instead of tuning the order of rules owned by two other services.