monitoring: alert on containers failing their healthcheck #394

Merged

mysticalsoap merged 1 commit from monitoring/unhealthy-alert into trunk

2026-10-07 23:14:41 -04:00

mysticalsoap commented

2026-10-07 23:07:45 -04:00

Owner

Problem

A container that is running but failing its own Docker healthcheck alerts nothing. The blackbox internal probes only cover HTTP liveness, and Monitoring.md says so. Closes #126.

Fix

No new exporter or socket proxy: cAdvisor v0.55.1 (already running) exports container_health_state for every container, and Prometheus already has it for all 80 running ones (60 at 1, 20 at -1 for no healthcheck). One alert rule, ContainerUnhealthy, on container_health_state{name!=""} == 0 for 6m, severity=critical, category=containers (routes to homelab-containers).

The 6m is because cAdvisor reports 0 for both unhealthy and starting, so the for: has to outlast the longest start_period in the stack (crowdsec, 300s). No autoheal, as the issue asked: a restart clears the signal and adds exec/restart volume on dockerd (#117). cAdvisor already runs one ContainerInspect per container per 30s housekeeping to get this, so nothing new hits the daemon.

Docs: Monitoring.md's "nothing scrapes or alerts on that today" replaced with a pointer to the rule, alert table row, ContainerUnhealthy added to the generic rules list in Adding-a-New-Service.md, Services.md cadvisor row.

Verification

promtool check rules in the running prometheus container: 39 rules, SUCCESS.
Edit via a sandboxed write changed the file's inode, so /-/reload loaded the old content (the documented bind-mount gotcha); docker restart prometheus picked it up.
Throwaway container health-probe (alpine:3, --health-cmd false --health-interval 5s --health-retries 1): docker inspect showed unhealthy within 12s, container_health_state{name="health-probe"} was 0 in Prometheus within a minute.
Alert went pending, then firing after the 6m window: pending at the first evaluation (02:54:43 UTC), firing at 03:00:43. The restarted prometheus container showed up pending alongside it (starting = 0) and dropped out after its first passing healthcheck, which is the case the 6m exists for.
Alertmanager received it and ntfy-alertmanager delivered to homelab-containers: alertmanager_notifications_total{integration="webhook"} went 3 → 4 with every _failed_total reason at 0, and ntfy's stats line went messages_published=385 → 386 in the same minute.
docker rm -f health-probe: inactive in Prometheus at the 03:06:43 evaluation, about 5m after removal. cAdvisor stamps its samples, so Prometheus writes no staleness marker and the series lingers for the 5m lookback (Prometheus docs § Staleness). Added as a Monitoring.md gotcha; a container that turns healthy again resolves on the next evaluation.

🤖 Generated with Claude Code

## Problem A container that is running but failing its own Docker healthcheck alerts nothing. The blackbox internal probes only cover HTTP liveness, and Monitoring.md says so. Closes #126. ## Fix No new exporter or socket proxy: cAdvisor v0.55.1 (already running) exports `container_health_state` for every container, and Prometheus already has it for all 80 running ones (60 at `1`, 20 at `-1` for no healthcheck). One alert rule, `ContainerUnhealthy`, on `container_health_state{name!=""} == 0` for 6m, `severity=critical`, `category=containers` (routes to `homelab-containers`). The 6m is because cAdvisor reports `0` for both `unhealthy` and `starting`, so the `for:` has to outlast the longest `start_period` in the stack (crowdsec, 300s). No autoheal, as the issue asked: a restart clears the signal and adds exec/restart volume on dockerd (#117). cAdvisor already runs one `ContainerInspect` per container per 30s housekeeping to get this, so nothing new hits the daemon. Docs: Monitoring.md's "nothing scrapes or alerts on that today" replaced with a pointer to the rule, alert table row, `ContainerUnhealthy` added to the generic rules list in Adding-a-New-Service.md, Services.md cadvisor row. ## Verification - `promtool check rules` in the running prometheus container: 39 rules, SUCCESS. - Edit via a sandboxed write changed the file's inode, so `/-/reload` loaded the old content (the documented bind-mount gotcha); `docker restart prometheus` picked it up. - Throwaway container `health-probe` (`alpine:3`, `--health-cmd false --health-interval 5s --health-retries 1`): `docker inspect` showed `unhealthy` within 12s, `container_health_state{name="health-probe"}` was `0` in Prometheus within a minute. - Alert went pending, then firing after the 6m window: pending at the first evaluation (02:54:43 UTC), firing at 03:00:43. The restarted prometheus container showed up pending alongside it (`starting` = 0) and dropped out after its first passing healthcheck, which is the case the 6m exists for. - Alertmanager received it and ntfy-alertmanager delivered to `homelab-containers`: `alertmanager_notifications_total{integration="webhook"}` went 3 → 4 with every `_failed_total` reason at 0, and ntfy's stats line went `messages_published=385` → `386` in the same minute. - `docker rm -f health-probe`: inactive in Prometheus at the 03:06:43 evaluation, about 5m after removal. cAdvisor stamps its samples, so Prometheus writes no staleness marker and the series lingers for the 5m lookback (Prometheus docs § Staleness). Added as a Monitoring.md gotcha; a container that turns healthy again resolves on the next evaluation. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added 1 commit

2026-10-07 23:07:45 -04:00

monitoring: alert on containers failing their healthcheck

validate / validate (pull_request) Successful in 25s

Details

77594f5799

cAdvisor exports container_health_state for every container, so the
exporter-plus-socket-proxy #126 planned is not needed: one rule on
container_health_state == 0 covers it.

cAdvisor reports 0 for "starting" as well as "unhealthy", so the rule
waits 6m, past crowdsec's 300s start_period, the longest in the stack.
Deliberately no autoheal: restarting an unhealthy container clears the
signal, and the exec/restart volume feeds the dockerd leak (#117).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap referenced this pull request

2026-10-07 23:08:07 -04:00

Alert on unhealthy containers #126

mysticalsoap merged commit e68d33c84b into trunk

2026-10-07 23:14:41 -04:00

mysticalsoap deleted branch monitoring/unhealthy-alert

2026-10-07 23:14:41 -04:00

mysticalsoap referenced this pull request from a commit

2026-10-07 23:14:43 -04:00

Merge pull request 'monitoring: alert on containers failing their healthcheck' (#394) from monitoring/unhealthy-alert into trunk