DNS watchdog has no flap damping: a failing primary costs a full cache flush every interval #166

New issue

Open

opened 2026-08-24 22:57:28 -04:00 by mysticalsoap · 0 comments

mysticalsoap commented

2026-08-24 22:57:28 -04:00

Owner

The #95 watchdog re-pins the primary whenever its direct probe succeeds while resolved's selection sits on the fallback — and repin_dns_primary ends with resolvectl flush-caches. When the primary is persistently unreachable from resolved's own sockets but reachable from the watchdog's (the #118 condition), that becomes a permanent loop: observed 2026-08-24, seven re-pins between 20:08 and 20:17, each wiping the entire host's DNS cache, each followed by resolved re-failing within seconds. Every application doing frequent lookups saw multi-second stalls plus a cold cache every ~90s — Discord through the tunnel was visibly in-and-out for exactly as long as a tunnel was up.

The re-pin exists to rescue split-horizon resolution after a transient primary blip; it should recognize when the rescue isn't taking. Shape: track consecutive re-pins that fail to stick (selection back off the primary by the next interval), back off the retry cadence sharply after a few, and log the give-up once at warning — a primary that never sticks is #118's bug, not something to fight every 15 seconds. The unconditional flush also deserves a look: it exists to drop stale public answers from the fallback window, but in a flap loop it is the dominant user-visible cost.

🤖 Generated with Claude Code

The #95 watchdog re-pins the primary whenever its direct probe succeeds while resolved's selection sits on the fallback — and `repin_dns_primary` ends with `resolvectl flush-caches`. When the primary is *persistently* unreachable from resolved's own sockets but reachable from the watchdog's (the #118 condition), that becomes a permanent loop: observed 2026-08-24, seven re-pins between 20:08 and 20:17, each wiping the entire host's DNS cache, each followed by resolved re-failing within seconds. Every application doing frequent lookups saw multi-second stalls plus a cold cache every ~90s — Discord through the tunnel was visibly in-and-out for exactly as long as a tunnel was up. The re-pin exists to rescue split-horizon resolution after a *transient* primary blip; it should recognize when the rescue isn't taking. Shape: track consecutive re-pins that fail to stick (selection back off the primary by the next interval), back off the retry cadence sharply after a few, and log the give-up once at warning — a primary that never sticks is #118's bug, not something to fight every 15 seconds. The unconditional flush also deserves a look: it exists to drop stale public answers from the fallback window, but in a flap loop it is the dominant user-visible cost. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added the

bug

label

2026-08-24 22:57:28 -04:00

mysticalsoap referenced this issue

2026-08-24 23:10:06 -04:00

Split-horizon recipe: a LAN-address primary should ride the tunnel link alone, without a public fallback #167