Host WAN visibility gap: node-exporter's network metrics are container-netns only #176

New issue

Closed

opened 2026-08-26 13:23:08 -04:00 by mysticalsoap · 1 comment

mysticalsoap commented

2026-08-26 13:23:08 -04:00

Owner

Hit while diagnosing aqomui#118 (tunnel UDP loss windows): node-exporter runs on a bridge network, so node_network_* only covers its own container eth0/lo — enp5s0 and tunnel interfaces were never collected, and no dashboard can answer 'was the uplink saturated at time X'. cadvisor covers per-container NICs (that's how gluetun egress history was recovered), but direct host egress (Jellyfin streams, host tunnels) is invisible; the Traefik access log was the only witness for public-service egress.

Fix candidates: run node-exporter with network_mode: host (the usual deployment for it — also fixes any other netns-scoped collectors like sockstat attribution), or a second minimal exporter on host network for network counters only. While at it, consider a persistent WAN-health probe that deliberately bypasses any host VPN (ICMP/UDP loss+latency to a fixed target from the physical interface) — every tunnel-loss investigation so far has had zero baseline for 'was the raw uplink healthy at that moment'.

🤖 Generated with Claude Code

Hit while diagnosing aqomui#118 (tunnel UDP loss windows): node-exporter runs on a bridge network, so `node_network_*` only covers its own container eth0/lo — enp5s0 and tunnel interfaces were never collected, and no dashboard can answer 'was the uplink saturated at time X'. cadvisor covers per-container NICs (that's how gluetun egress history was recovered), but direct host egress (Jellyfin streams, host tunnels) is invisible; the Traefik access log was the only witness for public-service egress. Fix candidates: run node-exporter with `network_mode: host` (the usual deployment for it — also fixes any other netns-scoped collectors like sockstat attribution), or a second minimal exporter on host network for network counters only. While at it, consider a persistent WAN-health probe that deliberately bypasses any host VPN (ICMP/UDP loss+latency to a fixed target from the physical interface) — every tunnel-loss investigation so far has had zero baseline for 'was the raw uplink healthy at that moment'. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap referenced this issue

2026-10-08 16:30:52 -04:00

No egress shaping: a saturated uplink collapses downstream throughput #156

mysticalsoap commented

2026-10-08 16:32:51 -04:00

Author

Owner

Value, what others do, plan

Value. Continuous host NIC counters for enp5s0 and any tunnel interface, with history.

Answers "was the uplink saturated at time X" in any tunnel-loss or CI-collapse investigation. So far that had to be reconstructed from cadvisor and the Traefik access log.
Makes #156 verifiable and alertable (uplink pinned at its ceiling for N minutes).
Fixes every other netns-scoped collector at once: sockstat, netstat (TCP retransmits, UDP errors), arp and conntrack currently describe the container's own near-empty namespace, so those dashboard panels show nothing real today.

What others do. node_exporter's README gives network_mode: host + pid: host + /:/host:ro,rslave as the container deployment, with "deploying in containers requires extra care in order to avoid monitoring the container itself". kube-prometheus's DaemonSet is hostNetwork: true, hostPID: true, listener bound to loopback behind a proxy, and --collector.netdev.device-exclude=^(veth.*|[a-f0-9]{15})$ plus the same regex for netclass. The prometheus.io guide runs the binary on the host. Host netns is the standard; the bridge placement here is the deviation.

Plan.

network_mode: host; drop the internal network attachment (compose rejects both).
--web.listen-address=172.19.1.1:9100, the monitoring_internal gateway: reachable from Prometheus, no new 0.0.0.0 listener, nothing on the LAN side.
Prometheus target becomes 172.19.1.1:9100. The dashboard selects instance through a $node variable from node_uname_info, so the rename is absorbed; no alert rule references the instance.
--collector.netdev.device-exclude=^(veth.*|br-.*|docker0)$ and the same for netclass.ignored-devices. ~150 veth and bridge interfaces today, each a full series set; cadvisor already covers per-container NICs.
cap_drop ALL and no-new-privileges unchanged. First host-network container in the stack; noted in Networking.md.

Verify. node_network_receive_bytes_total{device="enp5s0"} exists (and tun_aqomui while connected); node_sockstat_TCP_inuse matches ss -s on the host; ss -tln shows 9100 only on 172.19.1.1.

Not in this change. The WAN health probe that bypasses the host VPN. blackbox-exporter exists but sits on bridges and is captured by aqomui's /1 routes, so it needs its own aqomui bypass entry first. Separate issue once this lands.

## Value, what others do, plan **Value.** Continuous host NIC counters for enp5s0 and any tunnel interface, with history. - Answers "was the uplink saturated at time X" in any tunnel-loss or CI-collapse investigation. So far that had to be reconstructed from cadvisor and the Traefik access log. - Makes #156 verifiable and alertable (uplink pinned at its ceiling for N minutes). - Fixes every other netns-scoped collector at once: sockstat, netstat (TCP retransmits, UDP errors), arp and conntrack currently describe the container's own near-empty namespace, so those dashboard panels show nothing real today. **What others do.** node_exporter's README gives `network_mode: host` + `pid: host` + `/:/host:ro,rslave` as the container deployment, with "deploying in containers requires extra care in order to avoid monitoring the container itself". kube-prometheus's DaemonSet is `hostNetwork: true`, `hostPID: true`, listener bound to loopback behind a proxy, and `--collector.netdev.device-exclude=^(veth.*|[a-f0-9]{15})$` plus the same regex for netclass. The prometheus.io guide runs the binary on the host. Host netns is the standard; the bridge placement here is the deviation. **Plan.** - `network_mode: host`; drop the `internal` network attachment (compose rejects both). - `--web.listen-address=172.19.1.1:9100`, the monitoring_internal gateway: reachable from Prometheus, no new 0.0.0.0 listener, nothing on the LAN side. - Prometheus target becomes `172.19.1.1:9100`. The dashboard selects instance through a `$node` variable from node_uname_info, so the rename is absorbed; no alert rule references the instance. - `--collector.netdev.device-exclude=^(veth.*|br-.*|docker0)$` and the same for `netclass.ignored-devices`. ~150 veth and bridge interfaces today, each a full series set; cadvisor already covers per-container NICs. - cap_drop ALL and no-new-privileges unchanged. First host-network container in the stack; noted in Networking.md. **Verify.** `node_network_receive_bytes_total{device="enp5s0"}` exists (and `tun_aqomui` while connected); `node_sockstat_TCP_inuse` matches `ss -s` on the host; `ss -tln` shows 9100 only on 172.19.1.1. **Not in this change.** The WAN health probe that bypasses the host VPN. blackbox-exporter exists but sits on bridges and is captured by aqomui's /1 routes, so it needs its own aqomui bypass entry first. Separate issue once this lands.

mysticalsoap referenced this issue from a pull request that will close it,

2026-10-08 16:49:21 -04:00

monitoring: node-exporter in the host netns #412

mysticalsoap referenced this issue from a commit

2026-10-08 16:49:22 -04:00

monitoring: node-exporter in the host netns

mysticalsoap closed this issue

2026-10-08 16:53:38 -04:00

mysticalsoap referenced this issue

2026-10-08 17:02:23 -04:00

WAN health probe: ICMP loss and RTT from the physical link, bypassing the host VPN #413