monitoring: node-exporter in the host netns #412

Merged

mysticalsoap merged 1 commit from monitoring/node-exporter-host-netns into trunk

2026-10-08 16:53:38 -04:00

mysticalsoap commented

2026-10-08 16:49:21 -04:00

Owner

Problem

node-exporter sat on a bridge, so netdev, sockstat, netstat and arp described its own namespace. enp5s0 and tunnel interfaces never appeared in node_network_*, nothing could say whether the uplink was saturated at a given time, and #156 has no way to verify a shaper.

Fix

network_mode: host, as upstream's container deployment and kube-prometheus's DaemonSet do. Listener bound to the monitoring_internal gateway (172.19.1.1:9100) so the only new listener is one Prometheus can reach; the scrape lands on the host INPUT chain, so one ufw rule keyed on the pinned subnet, documented next to the existing port 53 case. veth, br-* and docker0 excluded from netdev and netclass: ~150 of them today, each a full series set, already covered per container by cadvisor.

The WAN health probe from the issue is deferred: blackbox-exporter is captured by aqomui's /1 routes and needs its own bypass entry first.

Verification

Live on this branch via just up monitoring and a Prometheus reload:

node_network_receive_bytes_total has enp5s0, tun_aqomui, wlan0, lo; zero veth series
ss -tln shows 9100 on 172.19.1.1 only
Prometheus target 172.19.1.1:9100 up after the ufw rule; rate(node_network_transmit_bytes_total{device="enp5s0"}[1m]) returns live values
textfile metrics (docker_backup_last_success_timestamp_seconds) still arrive
node_uname_info nodename is now the host's, so the dashboard's $node variable picks up the new instance
ufw rule mirrored into dotfiles by backup-configs

Closes #176

🤖 Generated with Claude Code

## Problem node-exporter sat on a bridge, so netdev, sockstat, netstat and arp described its own namespace. enp5s0 and tunnel interfaces never appeared in `node_network_*`, nothing could say whether the uplink was saturated at a given time, and #156 has no way to verify a shaper. ## Fix `network_mode: host`, as upstream's container deployment and kube-prometheus's DaemonSet do. Listener bound to the `monitoring_internal` gateway (`172.19.1.1:9100`) so the only new listener is one Prometheus can reach; the scrape lands on the host `INPUT` chain, so one ufw rule keyed on the pinned subnet, documented next to the existing port 53 case. veth, `br-*` and `docker0` excluded from netdev and netclass: ~150 of them today, each a full series set, already covered per container by cadvisor. The WAN health probe from the issue is deferred: blackbox-exporter is captured by aqomui's /1 routes and needs its own bypass entry first. ## Verification Live on this branch via `just up monitoring` and a Prometheus reload: - `node_network_receive_bytes_total` has `enp5s0`, `tun_aqomui`, `wlan0`, `lo`; zero `veth` series - `ss -tln` shows 9100 on `172.19.1.1` only - Prometheus target `172.19.1.1:9100` up after the ufw rule; `rate(node_network_transmit_bytes_total{device="enp5s0"}[1m])` returns live values - textfile metrics (`docker_backup_last_success_timestamp_seconds`) still arrive - node_uname_info nodename is now the host's, so the dashboard's `$node` variable picks up the new instance - ufw rule mirrored into dotfiles by `backup-configs` Closes #176 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added 1 commit

2026-10-08 16:49:21 -04:00

monitoring: node-exporter in the host netns

validate / validate (pull_request) Successful in 18s

Details

c75f54336c

On a bridge, netdev, sockstat, netstat and arp report the container's
own namespace: enp5s0 and tunnel interfaces never appeared in
node_network_*, so nothing could say whether the uplink was saturated
at a given time, and #156 has no way to verify a shaper. Upstream's
container deployment is network_mode: host plus pid: host.

Bound to the monitoring_internal gateway rather than 0.0.0.0 so the
only new listener is one Prometheus can reach. ufw's deny incoming
drops that hairpin, hence the rule in Public-Exposure.md. veth and
bridge devices are excluded: ~150 of them, each a full series set, all
already covered per container by cadvisor.

Refs #176

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap merged commit 0d9273b536 into trunk

2026-10-08 16:53:38 -04:00

mysticalsoap deleted branch monitoring/node-exporter-host-netns

2026-10-08 16:53:38 -04:00

mysticalsoap referenced this pull request from a commit

2026-10-08 16:53:38 -04:00

Merge pull request 'monitoring: node-exporter in the host netns' (#412) from monitoring/node-exporter-host-netns into trunk

mysticalsoap referenced this pull request

2026-10-08 17:02:23 -04:00

WAN health probe: ICMP loss and RTT from the physical link, bypassing the host VPN #413

mysticalsoap referenced this pull request

2026-10-08 17:24:04 -04:00