monitoring: ICMP probe of the WAN link from the physical interface #415

Merged

mysticalsoap merged 2 commits from monitoring/wan-icmp-probe into trunk

2026-10-08 17:56:55 -04:00

mysticalsoap commented

2026-10-08 17:55:26 -04:00

Owner

Problem

Host NIC counters (#412) say how much went through enp5s0; nothing said whether the raw uplink was healthy at the time. Every tunnel-loss investigation lacked that baseline (#413).

Fix

blackbox icmp_wan module pinging 1.1.1.1 and 9.9.9.9 every 15 s via a blackbox-icmp-wan job, labelled category: network. No capability added: Docker's default ping_group_range lets the prober use unprivileged ping sockets under cap_drop: ALL. A network alert group with WanPacketLoss (delivery rate over 10 m below 90%) and WanLatencyHigh (RTT over 100 ms for 10 m); BlackboxProbeFailed excludes the job so an ISP outage doesn't page as a monitoring fault. Monitoring.md § WAN probe documents the aqomui dependency: the probe measures the physical link only through the 172.18.7.0/24 icmp network bypass entry (aqomui#226).

Second commit: CONTRIBUTING's stale-inode note now covers files a merge pull changed, not only ones a branch added. Hit on prometheus.yml during this work.

Verification

Live on this branch (just recreate prometheus, blackbox reloaded):

Both targets up in Prometheus, probe_icmp_duration_seconds{phase="rtt"} ~12 ms, matching ping -I enp5s0 1.1.1.1 from the host.
promtool check rules: 45 rules; network group lists both new rules.
Path proof with aqomui connected and the icmp entry in place: iptables-legacy -t mangle -L aqomui_bypass_net -v -n shows the icmp rule's packet counter climbing (9, 95, 99 across three reads) alongside the udp 51820 rule. RTT alone can't prove it any more: with the shaper (#414) up, tunnel and raw RTT both sit near 12 ms.
Direct probe before the entry existed also succeeded, so the module works regardless; only the path differs.

Closes #413

🤖 Generated with Claude Code

## Problem Host NIC counters (#412) say how much went through enp5s0; nothing said whether the raw uplink was healthy at the time. Every tunnel-loss investigation lacked that baseline (#413). ## Fix blackbox `icmp_wan` module pinging `1.1.1.1` and `9.9.9.9` every 15 s via a `blackbox-icmp-wan` job, labelled `category: network`. No capability added: Docker's default `ping_group_range` lets the prober use unprivileged ping sockets under `cap_drop: ALL`. A `network` alert group with `WanPacketLoss` (delivery rate over 10 m below 90%) and `WanLatencyHigh` (RTT over 100 ms for 10 m); `BlackboxProbeFailed` excludes the job so an ISP outage doesn't page as a monitoring fault. Monitoring.md § WAN probe documents the aqomui dependency: the probe measures the physical link only through the `172.18.7.0/24 icmp` network bypass entry (aqomui#226). Second commit: CONTRIBUTING's stale-inode note now covers files a merge pull *changed*, not only ones a branch added. Hit on prometheus.yml during this work. ## Verification Live on this branch (`just recreate prometheus`, blackbox reloaded): - Both targets `up` in Prometheus, `probe_icmp_duration_seconds{phase="rtt"}` ~12 ms, matching `ping -I enp5s0 1.1.1.1` from the host. - `promtool check rules`: 45 rules; `network` group lists both new rules. - Path proof with aqomui connected and the icmp entry in place: `iptables-legacy -t mangle -L aqomui_bypass_net -v -n` shows the icmp rule's packet counter climbing (9, 95, 99 across three reads) alongside the udp 51820 rule. RTT alone can't prove it any more: with the shaper (#414) up, tunnel and raw RTT both sit near 12 ms. - Direct probe before the entry existed also succeeded, so the module works regardless; only the path differs. Closes #413 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added 2 commits

2026-10-08 17:55:26 -04:00

monitoring: ICMP probe of the WAN link from the physical interface 10ef67b988

Every tunnel-loss investigation lacked a baseline for whether the ISP
link itself was healthy. blackbox pings two anycast resolvers every
15 s; Docker's default ping_group_range makes the ICMP prober work as
an unprivileged ping socket, so cap_drop ALL stays.

The probe only sees the physical link through aqomui's icmp network
bypass entry for blackbox's subnet (aqomui#226); a bare-subnet entry
would route this container's traffic to other containers out enp5s0
as well. BlackboxProbeFailed excludes the job so an ISP outage lands
on the network category rather than paging as a monitoring fault.

Refs #413

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: a merge pull also re-inodes bind-mounted files it changed

validate / validate (pull_request) Successful in 21s

Details

e3a37718d8

Hit on prometheus.yml: changed by #412, pulled, then edited again on the
next branch; the container kept the pre-pull inode and the reload read
nothing new. Same fix as the added-file case.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap merged commit 06eedad23f into trunk

2026-10-08 17:56:55 -04:00

mysticalsoap deleted branch monitoring/wan-icmp-probe

2026-10-08 17:56:55 -04:00

mysticalsoap referenced this pull request from a commit

2026-10-08 17:56:55 -04:00

Merge pull request 'monitoring: ICMP probe of the WAN link from the physical interface' (#415) from monitoring/wan-icmp-probe into trunk