alt-DNS at a local address intermittently unreachable on the tunnel link #118

Open
opened 2026-08-22 12:59:51 -04:00 by mysticalsoap · 2 comments
Owner

Observed 2026-08-21 (live). With alt_dns1 pointing at this host's own resolver (192.168.0.205) applied to the tunnel link with ~., resolved's query traffic to the primary was observed taking 192.168.0.205 via 10.96.0.1 dev tun_aqomui — into the VPN, toward an exit that cannot route an RFC1918 destination. The primary therefore always failed, selection fell to the public fallback and stayed there (compounded by the watchdog bug), and split-horizon resolution was dead host-wide while any tunnel was up.

Observed 2026-08-22 (live, same config). Not reproducing: tunnel up, ~. on tun_aqomui, CurrentDNSServer: 192.168.0.205, split-horizon names answered correctly in ~5ms. Plain routing agrees it should work — the destination is the host's own address, so the local table (or the connected /24 for a true LAN resolver) outranks any tunnel default; only an oif-bound socket would be forced into the tunnel.

So the failure is conditional, not by-construction, and the trigger is unknown. Candidates worth ruling in/out: whether resolved binds link-scoped query sockets to the link's ifindex in some versions/paths and not others, a race at link setup before routes settle, or a difference between the code path that applied the DNS (aqomui set_dns vs OpenVPN dns-updown).

The #34 split-horizon recipe (alt-DNS = local resolver, public fallback second) depends on this path being reliable — when it degrades, the failure is silent public answers, indistinguishable from the services being down. Needs a reproduction before 0.9.1, or at minimum a documented diagnostic (resolvectl status <tun> | grep 'Current DNS').

**Observed 2026-08-21 (live).** With `alt_dns1` pointing at this host's own resolver (`192.168.0.205`) applied to the tunnel link with `~.`, resolved's query traffic to the primary was observed taking `192.168.0.205 via 10.96.0.1 dev tun_aqomui` — into the VPN, toward an exit that cannot route an RFC1918 destination. The primary therefore always failed, selection fell to the public fallback and stayed there (compounded by the watchdog bug), and split-horizon resolution was dead host-wide while any tunnel was up. **Observed 2026-08-22 (live, same config).** Not reproducing: tunnel up, `~.` on `tun_aqomui`, `CurrentDNSServer: 192.168.0.205`, split-horizon names answered correctly in ~5ms. Plain routing agrees it should work — the destination is the host's own address, so the `local` table (or the connected /24 for a true LAN resolver) outranks any tunnel default; only an oif-bound socket would be forced into the tunnel. **So the failure is conditional, not by-construction, and the trigger is unknown.** Candidates worth ruling in/out: whether resolved binds link-scoped query sockets to the link's ifindex in some versions/paths and not others, a race at link setup before routes settle, or a difference between the code path that applied the DNS (aqomui `set_dns` vs OpenVPN dns-updown). The #34 split-horizon recipe (alt-DNS = local resolver, public fallback second) depends on this path being reliable — when it degrades, the failure is silent public answers, indistinguishable from the services being down. Needs a reproduction before 0.9.1, or at minimum a documented diagnostic (`resolvectl status <tun> | grep 'Current DNS'`).
Author
Owner

Reproduced hard on 2026-08-24, all evening, with a strong new signal. Every tunnel-up shows the same ladder in resolved's journal: ~7s after aqomui sets the tun link's DNS, 'Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.0.205', then failover to 9.9.9.9; the #95 watchdog's direct probe (unbound socket, LAN route) succeeds, re-pins, and resolved fails again — seven flips between 20:08 and 20:17, each with a full flush-caches. Same pattern at 17:50–18:00 and 22:47. The 'grace period over, resuming full feature set' entries at 18:01:22 and 20:17:40 are followed by an immediate re-degrade, so the path to .205 is consistently dead from resolved's sockets while the tunnel is up, consistently fine from the watchdog's.

That asymmetry is the bound-socket candidate from the issue: resolved binding link-scoped queries to tun_aqomui's ifindex would force RFC1918-bound packets into the tunnel (no route can rescue an oif-bound socket), while the watchdog probes unbound. Next tunnel session, two captures settle it: 'tcpdump -ni tun_aqomui udp port 53' vs 'tcpdump -ni enp5s0 udp port 53 and host 192.168.0.205' — if resolved's queries to .205 appear on tun and never on enp5s0, the mechanism is confirmed and a per-link LAN-address primary can't work by construction.

User-visible cost meanwhile: the flap loop caused multi-second resolution stalls plus a cold cache every ~90s — surfaced as Discord dropping in and out through the tunnel. The watchdog's contribution (no flap damping, unconditional flush) split out as its own issue.

🤖 Generated with Claude Code

Reproduced hard on 2026-08-24, all evening, with a strong new signal. Every tunnel-up shows the same ladder in resolved's journal: ~7s after aqomui sets the tun link's DNS, 'Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.0.205', then failover to 9.9.9.9; the #95 watchdog's direct probe (unbound socket, LAN route) succeeds, re-pins, and resolved fails again — seven flips between 20:08 and 20:17, each with a full flush-caches. Same pattern at 17:50–18:00 and 22:47. The 'grace period over, resuming full feature set' entries at 18:01:22 and 20:17:40 are followed by an immediate re-degrade, so the path to .205 is consistently dead from resolved's sockets while the tunnel is up, consistently fine from the watchdog's. That asymmetry is the bound-socket candidate from the issue: resolved binding link-scoped queries to tun_aqomui's ifindex would force RFC1918-bound packets into the tunnel (no route can rescue an oif-bound socket), while the watchdog probes unbound. Next tunnel session, two captures settle it: 'tcpdump -ni tun_aqomui udp port 53' vs 'tcpdump -ni enp5s0 udp port 53 and host 192.168.0.205' — if resolved's queries to .205 appear on tun and never on enp5s0, the mechanism is confirmed and a per-link LAN-address primary can't work by construction. User-visible cost meanwhile: the flap loop caused multi-second resolution stalls plus a cold cache every ~90s — surfaced as Discord dropping in and out through the tunnel. The watchdog's contribution (no flap damping, unconditional flush) split out as its own issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
Owner

Comparative finding from the Proton desktop app (its custom-DNS setting ran this exact split-horizon config 'flawlessly', per live use): reading the installed app source (proton-vpn-gtk-app 4.16.5, both the WG and OpenVPN NM backends), custom DNS puts the server on the tunnel connection as dns=[192.168.0.205] with dns-priority=-1500 and ignore-auto-dns — alone, with no fallback server, and no explicit search domain in the custom branch. So the app also served .205 from the tunnel link, and it worked.

Two consequences. First, the intermittent-unreachability trigger can't be 'per-link LAN-address DNS never works' — the app era plus the 8-22 observation both show it working. Second, the pair-vs-single difference explains the felt contrast: with one server, resolved's sticky failover has nowhere to go — a blip is a failed query and a same-server retry, self-healing and invisible. Our [.205, 9.9.9.9] pair is what converts a blip into strand→re-pin→flush→flap.

Also relevant: the app's flawless run was WireGuard; aqomui runs OpenVPN+DCO, which has prior fwmark-inheritance history on this machine — protocol is back on the candidate list for the intermittent loss. The tcpdump discriminator from the previous comment still decides it.

Host resolver topology, confirmed while investigating: /etc/resolv.conf is NM-written pointing at 127.0.0.53 and nsswitch has resolve [!UNAVAIL=return] — every resolver path on the host goes through systemd-resolved, so per-link behavior governs all applications.

🤖 Generated with Claude Code

Comparative finding from the Proton desktop app (its custom-DNS setting ran this exact split-horizon config 'flawlessly', per live use): reading the installed app source (proton-vpn-gtk-app 4.16.5, both the WG and OpenVPN NM backends), custom DNS puts the server on the tunnel connection as dns=[192.168.0.205] with dns-priority=-1500 and ignore-auto-dns — **alone, with no fallback server**, and no explicit search domain in the custom branch. So the app also served .205 from the tunnel link, and it worked. Two consequences. First, the intermittent-unreachability trigger can't be 'per-link LAN-address DNS never works' — the app era plus the 8-22 observation both show it working. Second, the pair-vs-single difference explains the felt contrast: with one server, resolved's sticky failover has nowhere to go — a blip is a failed query and a same-server retry, self-healing and invisible. Our [.205, 9.9.9.9] pair is what converts a blip into strand→re-pin→flush→flap. Also relevant: the app's flawless run was WireGuard; aqomui runs OpenVPN+DCO, which has prior fwmark-inheritance history on this machine — protocol is back on the candidate list for the intermittent loss. The tcpdump discriminator from the previous comment still decides it. Host resolver topology, confirmed while investigating: /etc/resolv.conf is NM-written pointing at 127.0.0.53 and nsswitch has resolve [!UNAVAIL=return] — every resolver path on the host goes through systemd-resolved, so per-link behavior governs all applications. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
mysticalsoap/aqomui#118
No description provided.