change: service watches the network itself (#111)

mysticalsoap commented

2026-08-22 23:00:45 -04:00

Owner

Problem

monitor.NetMon and the network-change policy (aqomui_gui.network_change -- kill tunnels, reconnect, rebuild the bypass) ran as unprivileged gui code. With no gui process around, nobody reacted to a network change at all. Network policy belongs where tunnel state lives. Successor to #89. Closes #111.

Fix

New aqomui/observer.py: the utils network cluster (default_routes, default_interface, preferred_interface) moves in, plus NetObserver -- the ported NetMon poll as a GuardedThread the service owns.
The service reacts itself on transition: up = save dns, disconnect main+bypass, reconcile_bypass; down = disconnect both. Then it broadcasts the outcome (new D-Bus signal net_state_changed, snapshot method get_net_state).
The gui keeps only its share: latency pings and connect_last_server -- the reconnect is the user's autoconnect policy, so it stays client-side (a Qt-free shared class the cli can also use is follow-up work, see the issue filed alongside this PR). Teardown feedback reaches the ui through the existing status signals. log_from_thread lost its last consumer and is gone.

D-Bus surface changed: gui and service must upgrade together (same as #110).

Behavior deltas to be aware of:

A service (re)start with the network up runs the up reaction once (~2s in) and broadcasts -- so a running gui with autoconnect on now reconnects after a service restart (before: stayed disconnected). Reads as the declared-state semantics of autoconnect, but it is a change.
A transient /sys/class/net read error keeps the prior state instead of resetting to down (NetMon used to replay the full up reaction when the read recovered).

Verification

pytest: 305 passed, including new tests/test_observer.py (moved network-fact tests + poll_once transition cases) and reaction-policy tests for network_changed (service) and net_state_changed (gui).
CI lint gate clean locally.
Live round pending before merge (per the process rule): build-branch-and-install.sh, then (1) gui closed: unplug/replug the link, watch the service log react and the bypass rebuild; (2) gui open with autoconnect: same flip, expect teardown statuses, pings, reconnect; (3) service restart under a running gui: expect resync + the new auto-reconnect.

## Problem `monitor.NetMon` and the network-change policy (`aqomui_gui.network_change` -- kill tunnels, reconnect, rebuild the bypass) ran as unprivileged gui code. With no gui process around, nobody reacted to a network change at all. Network policy belongs where tunnel state lives. Successor to #89. Closes #111. ## Fix - New `aqomui/observer.py`: the utils network cluster (`default_routes`, `default_interface`, `preferred_interface`) moves in, plus `NetObserver` -- the ported NetMon poll as a `GuardedThread` the service owns. - The service reacts itself on transition: up = save dns, disconnect main+bypass, `reconcile_bypass`; down = disconnect both. Then it broadcasts the outcome (new D-Bus signal `net_state_changed`, snapshot method `get_net_state`). - The gui keeps only its share: latency pings and `connect_last_server` -- the reconnect is the user's autoconnect policy, so it stays client-side (a Qt-free shared class the cli can also use is follow-up work, see the issue filed alongside this PR). Teardown feedback reaches the ui through the existing status signals. `log_from_thread` lost its last consumer and is gone. **D-Bus surface changed: gui and service must upgrade together** (same as #110). Behavior deltas to be aware of: - A service (re)start with the network up runs the up reaction once (~2s in) and broadcasts -- so a running gui with autoconnect on now reconnects after a service restart (before: stayed disconnected). Reads as the declared-state semantics of autoconnect, but it is a change. - A transient `/sys/class/net` read error keeps the prior state instead of resetting to down (NetMon used to replay the full up reaction when the read recovered). ## Verification - `pytest`: 305 passed, including new `tests/test_observer.py` (moved network-fact tests + `poll_once` transition cases) and reaction-policy tests for `network_changed` (service) and `net_state_changed` (gui). - CI lint gate clean locally. - **Live round pending before merge** (per the process rule): `build-branch-and-install.sh`, then (1) gui closed: unplug/replug the link, watch the service log react and the bypass rebuild; (2) gui open with autoconnect: same flip, expect teardown statuses, pings, reconnect; (3) service restart under a running gui: expect resync + the new auto-reconnect.

mysticalsoap added 1 commit

2026-08-22 23:00:45 -04:00

change: service watches the network itself (#111 )

ci / test (pull_request) Successful in 29s

Details

824444ea28

NetMon and the reaction to what it saw -- kill tunnels, reconnect,
rebuild the bypass -- ran as unprivileged gui code, so with no gui
process around nobody reacted to a network change at all. The watcher
(NetObserver, a GuardedThread) and the reaction now live in the
service: save dns, tear down tunnels, rebuild the bypass, then
broadcast the outcome over net_state_changed. Clients keep only their
own share -- latency pings, and the reconnect, which is the user's
autoconnect policy, not the service's.

The observer starts from "down", so a service (re)start with the
network already up runs the up reaction once -- a boot with no gui
still gets its dns saved and the bypass rebuilt. A gui connecting
later replays the service's current state via get_net_state; startup
autoconnect rides on that now that no synthetic NetMon up-event
exists. A transient /sys read keeps the prior state instead of
resetting to "down", which used to replay the whole up reaction once
the read recovered.

The utils network cluster (default_routes, default_interface,
preferred_interface) moves into observer.py with the code built on it;
a standalone netinfo.py would have been a third home for it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysticalsoap commented

2026-08-22 23:37:11 -04:00

Author

Owner

Live round, first pass (service restart under a running gui, then gui relaunch):

Mechanism verified working. At 23:22:39 the restarted service reconciled, the old gui's owner-watch resynced, and at 23:22:40 the observer fired its first up-transition and ran the full reaction with no gui involvement (Network: new connection detected → dns saved, disconnects, bypass rebuilt, broadcast). The relaunched gui picked up state via get_net_state (Detected new network connection at 23:23:57).

The failed connect that followed was environmental, not this PR. The manual connect at 23:24:55 took 12s to reach Peer Connection Initiated (historically ~100ms to the same server, incl. earlier the same day) and the pre-existing 15s gui watchdog killed it 3s short of completing. Measured during diagnosis: 15% packet loss and ~1.1s RTT to the vpn server over the physical link -- uplink bufferbloat from torrent upload on the ~43Mbps uplink. A reproduced connect over D-Bus completed at 12.8s under the same congestion. The connect path is untouched by this diff.

Observed, accepted for now: a restart with the network up runs three bypass rebuilds in ~1.5s (init reconcile_bypass → gui resync bypass call → observer up-reaction reconcile). Each carries a recovery case the others don't (no-gui restarts / owner registration when the on-disk cgroup is gone / networks that come up later), all are idempotent, so deduplicating risks the #84 class of regression for log noise. Flagging rather than fixing.

Still pending before merge: link flip with gui closed; link flip with gui open and autoconnect=1 (note: current config has autoconnect: 0, so no reconnect is the expected result of a flip test until it's enabled).

Live round, first pass (service restart under a running gui, then gui relaunch): **Mechanism verified working.** At 23:22:39 the restarted service reconciled, the old gui's owner-watch resynced, and at 23:22:40 the observer fired its first up-transition and ran the full reaction with no gui involvement (`Network: new connection detected` → dns saved, disconnects, bypass rebuilt, broadcast). The relaunched gui picked up state via `get_net_state` (`Detected new network connection` at 23:23:57). **The failed connect that followed was environmental, not this PR.** The manual connect at 23:24:55 took 12s to reach Peer Connection Initiated (historically ~100ms to the same server, incl. earlier the same day) and the pre-existing 15s gui watchdog killed it 3s short of completing. Measured during diagnosis: 15% packet loss and ~1.1s RTT to the vpn server over the physical link -- uplink bufferbloat from torrent upload on the ~43Mbps uplink. A reproduced connect over D-Bus completed at 12.8s under the same congestion. The connect path is untouched by this diff. **Observed, accepted for now: a restart with the network up runs three bypass rebuilds in ~1.5s** (init `reconcile_bypass` → gui resync `bypass` call → observer up-reaction reconcile). Each carries a recovery case the others don't (no-gui restarts / owner registration when the on-disk cgroup is gone / networks that come up later), all are idempotent, so deduplicating risks the #84 class of regression for log noise. Flagging rather than fixing. Still pending before merge: link flip with gui closed; link flip with gui open and `autoconnect=1` (note: current config has `autoconnect: 0`, so no reconnect is the *expected* result of a flip test until it's enabled).

mysticalsoap force-pushed change/network-observer from 824444ea28

ci / test (pull_request) Successful in 29s

Details

to 4d9cdfe737

ci / test (pull_request) Successful in 29s

Details

2026-08-23 00:26:48 -04:00

Compare

mysticalsoap commented

2026-08-23 00:27:15 -04:00

Author

Owner

Live flip test caught a real one: the link flip produced no reaction at all -- no down, no up, nothing logged. Root cause is inherited, not a port mistake: NetMon's "any interface operstate-up" check can never see a link flip on this host, because ~130 docker veths/bridges are permanently operstate-up -- and so is a connected tunnel's own DCO device. By interface state this machine never goes down, so the reaction code was unreachable. The 2019 design predates both containers-everywhere and DCO devices that outlive a link drop.

Reworked the watcher (amended into the branch commit, force-pushed): NetObserver now polls default_routes() and reports transitions of the default route -- gone, back, or changed (the changed-gateway case is one interface state could never see). Two identical polls required before reporting, so v4/v6 arriving seconds apart on reconnect collapse into one reaction instead of two teardown-rebuild cycles. default_routes() now skips the tunnels' own devices (tun_aqomui*/wg_aqomui*), so a plain redirect-gateway route can't read as a new network and react the tunnel to death -- this also hardens the bypass's use of the same helper. New tests cover all transition shapes + the filter; 308 passing.

Needs a rebuild + service restart, then the flip test again. Expected: Network: connection lost - closing tunnels within ~4s of the drop (2s poll x 2-poll stabilization), teardown of both tunnels, then on reconnect Network: new connection detected, bypass rebuild, broadcast, gui autoconnect.

Live flip test caught a real one: **the link flip produced no reaction at all** -- no down, no up, nothing logged. Root cause is inherited, not a port mistake: NetMon's "any interface operstate-up" check can never see a link flip on this host, because ~130 docker veths/bridges are permanently operstate-up -- and so is a connected tunnel's own DCO device. By interface state this machine never goes down, so the reaction code was unreachable. The 2019 design predates both containers-everywhere and DCO devices that outlive a link drop. Reworked the watcher (amended into the branch commit, force-pushed): NetObserver now polls `default_routes()` and reports transitions of the **default route** -- gone, back, or *changed* (the changed-gateway case is one interface state could never see). Two identical polls required before reporting, so v4/v6 arriving seconds apart on reconnect collapse into one reaction instead of two teardown-rebuild cycles. `default_routes()` now skips the tunnels' own devices (`tun_aqomui*`/`wg_aqomui*`), so a plain `redirect-gateway` route can't read as a new network and react the tunnel to death -- this also hardens the bypass's use of the same helper. New tests cover all transition shapes + the filter; 308 passing. Needs a rebuild + service restart, then the flip test again. Expected: `Network: connection lost - closing tunnels` within ~4s of the drop (2s poll x 2-poll stabilization), teardown of both tunnels, then on reconnect `Network: new connection detected`, bypass rebuild, broadcast, gui autoconnect.

mysticalsoap added 1 commit

2026-08-23 01:05:45 -04:00

fix: skip the bypass rebuild when no network is left

ci / test (pull_request) Successful in 33s

Details

ci / test (push) Successful in 39s

Details

faa4863f5a

The bypass tunnel's teardown asks cgroup_vpn for a rebuild, and the
service reacting to network loss (#111) now exercises that on every
outage: with the routes gone it built a cgroup against interface
"None" -- route and rp_filter errors, and a dnsmasq that could not
join it. apply_bypass already refuses to build without a default
route; cgroup_vpn gets the same guard, and the up reaction reconciles
the bypass when a network returns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysticalsoap commented

2026-08-23 01:05:55 -04:00

Author

Owner

Flip test round 2: the headline path works end to end. 01:03:39 Network: connection lost - closing tunnels -- the service detected the drop itself within ~2-4s and killed both tunnels; 01:03:49 Network: new connection detected -- dns saved, bypass rebuilt for enp5s0, broadcast, and the gui autoconnected main + bypass (NJ up in 1.5s, VA right behind). Detection, teardown, rebuild, broadcast, client reconnect all verified live.

One adjacent hole surfaced and is fixed in a second commit (faa4863): the bypass tunnel's teardown asks cgroup_vpn for a rebuild, which the network-loss reaction now triggers on every outage -- mid-outage the routes are gone, so it built a cgroup against interface "None" (route/rp_filter errors, dnsmasq unable to join). It now gets the same no-network guard apply_bypass already had; the up reaction reconciles when a network returns. Regression test added (309 passing).

Remaining before merge: one more flip to see the guard's Bypass: no network - rebuild deferred line replace the error spam on the down side.

**Flip test round 2: the headline path works end to end.** 01:03:39 `Network: connection lost - closing tunnels` -- the service detected the drop itself within ~2-4s and killed both tunnels; 01:03:49 `Network: new connection detected` -- dns saved, bypass rebuilt for enp5s0, broadcast, and the gui autoconnected main + bypass (NJ up in 1.5s, VA right behind). Detection, teardown, rebuild, broadcast, client reconnect all verified live. One adjacent hole surfaced and is fixed in a second commit (`faa4863`): the bypass tunnel's teardown asks `cgroup_vpn` for a rebuild, which the network-loss reaction now triggers on every outage -- mid-outage the routes are gone, so it built a cgroup against interface "None" (route/rp_filter errors, dnsmasq unable to join). It now gets the same no-network guard `apply_bypass` already had; the up reaction reconciles when a network returns. Regression test added (309 passing). Remaining before merge: one more flip to see the guard's `Bypass: no network - rebuild deferred` line replace the error spam on the down side.

mysticalsoap merged commit faa4863f5a into trunk

2026-08-23 01:12:22 -04:00

Rows
Columns

change: service watches the network itself (#111) #121

Problem

Fix

Verification