fix: return DNS selection to the primary once it answers again #96

Merged

mysticalsoap merged 1 commit from fix/dns-fallback-watchdog into trunk

2026-08-20 16:36:41 -04:00

mysticalsoap commented

2026-08-20 15:41:54 -04:00

Owner

Problem

#95: systemd-resolved abandons a failed DNS server but never returns while the substitute keeps answering, and re-applying an identical server list does not reset its selection. A one-second blip during a ping-restart reconnect stranded the system on the fallback indefinitely — observed live twice in one afternoon while verifying #34. For a split-horizon primary that failure is silent: internal names resolve to their public addresses from then on, nothing looks broken, and there is no obvious way to notice or recover.

Fix

A watchdog thread per tunnel while an aqomui-applied DNS pair is up: read resolved's current server for the link over D-Bus (CurrentDNSServer on the resolve1 Link — no text parsing), and when it is not the primary, send one UDP probe straight at the primary. If it answers, force re-selection: set the primary alone, restore the pair, flush the answers the fallback served. Worst case is one 15s check interval on the fallback instead of forever.
Lifecycle: started from tunnel_up (OpenVPN main role) and the WireGuard connect path; exits on its own when the link disappears; each CONNECTED starts one if none is running, so it survives ping-restarts whether or not the device blinks. Explicitly stopped when the OpenVPN process exits.
The bypass dnsmasq gets --strict-order: slot order is priority, not a pool, so its favour-the-working-server logic can't strand bypass lookups on the fallback either.
The plain-resolv.conf path needs nothing — glibc already tries nameservers in listed order on every query.

Verification

141 passed (130 existing + 11 new): probe wire format, probe against a fake UDP resolver (reply / mangled-ID / silence), the exact repin command sequence, and the watchdog decision flow (primary-in-charge no-op, stuck-with-primary-up repins, primary-really-down stays, dead link exits).
Building blocks verified live against the running host: current_dns_server("tun_aqomui") returns the correct current server via D-Bus; probes at the real AdGuard and Quad9 answer, TEST-NET times out, a missing link returns None.
The repin sequence itself is the same changed-list trick proven by hand twice during the #34/#95 investigation (an identical-list re-apply demonstrably does not reset selection; primary-alone-then-pair does).
Not yet exercised end-to-end: the running service is on pre-fix code, so the full induce-a-blip → auto-recover loop should be watched once after install — the DNS: ... answers again - selection returned from ... log line marks it firing.

Closes #95

**Problem** #95: systemd-resolved abandons a failed DNS server but never returns while the substitute keeps answering, and re-applying an identical server list does not reset its selection. A one-second blip during a ping-restart reconnect stranded the system on the fallback indefinitely — observed live twice in one afternoon while verifying #34. For a split-horizon primary that failure is silent: internal names resolve to their public addresses from then on, nothing looks broken, and there is no obvious way to notice or recover. **Fix** - A watchdog thread per tunnel while an aqomui-applied DNS pair is up: read resolved's current server for the link over D-Bus (`CurrentDNSServer` on the resolve1 Link — no text parsing), and when it is not the primary, send one UDP probe straight at the primary. If it answers, force re-selection: set the primary alone, restore the pair, flush the answers the fallback served. Worst case is one 15s check interval on the fallback instead of forever. - Lifecycle: started from `tunnel_up` (OpenVPN main role) and the WireGuard connect path; exits on its own when the link disappears; each CONNECTED starts one if none is running, so it survives ping-restarts whether or not the device blinks. Explicitly stopped when the OpenVPN process exits. - The bypass dnsmasq gets `--strict-order`: slot order is priority, not a pool, so its favour-the-working-server logic can't strand bypass lookups on the fallback either. - The plain-resolv.conf path needs nothing — glibc already tries nameservers in listed order on every query. **Verification** - 141 passed (130 existing + 11 new): probe wire format, probe against a fake UDP resolver (reply / mangled-ID / silence), the exact repin command sequence, and the watchdog decision flow (primary-in-charge no-op, stuck-with-primary-up repins, primary-really-down stays, dead link exits). - Building blocks verified live against the running host: `current_dns_server("tun_aqomui")` returns the correct current server via D-Bus; probes at the real AdGuard and Quad9 answer, TEST-NET times out, a missing link returns None. - The repin sequence itself is the same changed-list trick proven by hand twice during the #34/#95 investigation (an identical-list re-apply demonstrably does not reset selection; primary-alone-then-pair does). - Not yet exercised end-to-end: the running service is on pre-fix code, so the full induce-a-blip → auto-recover loop should be watched once after install — the `DNS: ... answers again - selection returned from ...` log line marks it firing. Closes #95

mysticalsoap added 1 commit

2026-08-20 15:41:54 -04:00

fix: return DNS selection to the primary once it answers again

ci / test (pull_request) Successful in 48s

Details

2cc6578cdf

systemd-resolved abandons a failed server but never returns while the
substitute keeps answering, and re-applying an identical server list
does not reset its selection - so a one-second blip during a tunnel
reconnect stranded the system on the fallback indefinitely. For a
split-horizon primary that is silent data corruption: internal names
resolve to their public addresses from then on (#95).

A watchdog thread now runs while a tunnel with an aqomui-applied DNS
pair is up: when resolved's current server (read via the resolve1 D-Bus
CurrentDNSServer property, no text parsing) is not the primary, one UDP
probe goes straight at the primary, and if it answers, selection is
forced back by setting the primary alone, restoring the pair, and
flushing the answers the fallback served. Worst case is one check
interval on the fallback instead of forever. The watchdog exits with
its link and each CONNECTED starts one if none is running, so it rides
out ping-restarts either way.

The bypass dnsmasq gets --strict-order for the same reason: slot order
is priority, not a pool. The plain-resolv.conf path already tries
nameservers in listed order per query and needs nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysticalsoap commented

2026-08-20 15:50:09 -04:00

Author

Owner

Design note — polling vs events, tested live: resolved emits PropertiesChanged only when the configured server list changes (Manager-level DNS property). An induced selection switch (dead TEST-NET primary → fallback) fired no signal at all: CurrentDNSServer changes silently, so there is nothing to subscribe to and a periodic peek is the only detection. The idle tick is one local D-Bus property read (no DNS traffic); the UDP probe only runs while the fallback is in use. The idle interval bounds the worst-case time on the fallback, which is why it stays at 15s.

Design note — polling vs events, tested live: resolved emits PropertiesChanged only when the configured server *list* changes (Manager-level `DNS` property). An induced selection switch (dead TEST-NET primary → fallback) fired no signal at all: `CurrentDNSServer` changes silently, so there is nothing to subscribe to and a periodic peek is the only detection. The idle tick is one local D-Bus property read (no DNS traffic); the UDP probe only runs while the fallback is in use. The idle interval bounds the worst-case time on the fallback, which is why it stays at 15s.

mysticalsoap referenced this pull request from a commit

2026-08-20 15:58:30 -04:00

docs: note split-horizon DNS handling on the alternative-DNS setting

mysticalsoap referenced this pull request

2026-08-20 15:58:38 -04:00

docs: document the split-horizon DNS setup #94

mysticalsoap merged commit 84bd9b108e into trunk

2026-08-20 16:36:41 -04:00

mysticalsoap deleted branch fix/dns-fallback-watchdog

2026-08-20 16:36:41 -04:00