dockerd goroutine leak: mitigations #117

New issue

Open

opened 2026-08-17 15:07:10 -04:00 by mysticalsoap · 0 comments

mysticalsoap commented

2026-08-17 15:07:10 -04:00

Owner

Parked pending the upstream bug. Mitigations, roughly in order of value:

Loosen the default healthcheck interval fleet-wide (10s -> 30s or 60s). Doesn't fix the bug, but cuts exec volume 3-6x and extends safe uptime proportionally. 10s precision was never buying anything for depends_on gating, which just needs "ready within a reasonable window."
Expose dockerd's own Prometheus metrics (metrics-addr in daemon.json, not currently set) and alert on a go_goroutines threshold, so this pages before it cascades into "everything looks down."
Scheduled systemctl restart docker during low-traffic hours -- a pragmatic cap on goroutine accumulation, same category as the existing backup/maintenance timers. Blocked in practice by the Authelia/Traefik restart bug.
(Lower priority) Audit for healthchecks that could relax to condition: service_started where readiness-gating isn't actually needed.

Parked pending the upstream bug. Mitigations, roughly in order of value: - [ ] Loosen the default healthcheck interval fleet-wide (10s -> 30s or 60s). Doesn't fix the bug, but cuts exec volume 3-6x and extends safe uptime proportionally. 10s precision was never buying anything for `depends_on` gating, which just needs "ready within a reasonable window." - [ ] Expose dockerd's own Prometheus metrics (`metrics-addr` in `daemon.json`, not currently set) and alert on a `go_goroutines` threshold, so this pages before it cascades into "everything looks down." - [ ] Scheduled `systemctl restart docker` during low-traffic hours -- a pragmatic cap on goroutine accumulation, same category as the existing backup/maintenance timers. Blocked in practice by the Authelia/Traefik restart bug. - [ ] (Lower priority) Audit for healthchecks that could relax to `condition: service_started` where readiness-gating isn't actually needed.

mysticalsoap added the

bug

on-hold

labels

2026-08-17 15:07:10 -04:00

mysticalsoap referenced this issue

2026-08-17 17:49:56 -04:00

Alert on unhealthy containers #126