Alert on unhealthy containers #126

Open
opened 2026-08-17 17:49:56 -04:00 by mysticalsoap · 0 comments
Owner

Monitoring.md already admits it: "a failing Docker healthcheck marks a container unhealthy but nothing scrapes or alerts on that today." The blackbox internal probes cover HTTP apps; a container that is running-but-unhealthy by its own healthcheck's definition (auth failure inside, dead worker, full disk) alerts nothing.

House-pattern fix: a docker-state exporter behind its own GET-only socket-proxy (/containers/json only — same shape as promtail-socket-proxy), plus one alert on unhealthy. Explicitly not autoheal — restarting an unhealthy container masks the signal, and the goroutine-leak history (#117) argues against adding recurring exec/restart volume.

Monitoring.md already admits it: "a failing Docker healthcheck marks a container unhealthy but nothing scrapes or alerts on that today." The blackbox internal probes cover HTTP apps; a container that is running-but-unhealthy by its own healthcheck's definition (auth failure inside, dead worker, full disk) alerts nothing. House-pattern fix: a docker-state exporter behind its own GET-only socket-proxy (`/containers/json` only — same shape as promtail-socket-proxy), plus one alert on `unhealthy`. Explicitly not autoheal — restarting an unhealthy container masks the signal, and the goroutine-leak history (#117) argues against adding recurring exec/restart volume.
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
mysticalsoap/docker#126
No description provided.