dockerd cold-boot pile-up: overlapping image-usage walks saturate the daemon #182

New issue

Open

opened 2026-08-27 11:56:44 -04:00 by mysticalsoap · 0 comments

mysticalsoap commented

2026-08-27 11:56:44 -04:00

Owner

Second, separate bug found 2026-08-10 while chasing the goroutine leak (#181) — don't conflate them.

After a cold boot, dockerd climbed to ~31k goroutines within 20min and the whole fleet went unhealthy for ~50min. Captured dump showed zero leak signature — instead 14k+ goroutines in containerd/core/images.Walk doing per-image disk-usage calculation. Root cause from source: ImageService.Images() bounds each call internally (NumCPU*2 workers) but nothing guards against overlapping top-level calls — when one call takes longer than the interval between callers (cold caches + 108GB of images at the time), capped batches stack ~300 deep. cAdvisor's 30s housekeeping is the suspected repeat caller, unconfirmed (logs were empty). During the pile-up even docker images times out; lightweight socket endpoints (/info, /_ping) stay fast — diagnose with those, don't add heavy CLI calls to the backlog.

Mitigated since 2026-08-10 by the daily unused-image prune (dotfiles docker-image-prune.timer, -a -f --filter until=24h): 213→84 images, 108→37GB, and no recurrence on subsequent reboots. Remaining work:

confirm whether it still reproduces on cold boot at current image volume; if yes, pin the repeat caller (cAdvisor housekeeping interval is tunable)
consider upstream report: overlapping-call stacking in ImageService.Images() is arguably a moby bug (no request coalescing/backpressure on an expensive endpoint)

Second, separate bug found 2026-08-10 while chasing the goroutine leak (#181) — don't conflate them. After a cold boot, dockerd climbed to ~31k goroutines within 20min and the whole fleet went unhealthy for ~50min. Captured dump showed zero leak signature — instead 14k+ goroutines in `containerd/core/images.Walk` doing per-image disk-usage calculation. Root cause from source: `ImageService.Images()` bounds each call internally (`NumCPU*2` workers) but nothing guards against overlapping top-level calls — when one call takes longer than the interval between callers (cold caches + 108GB of images at the time), capped batches stack ~300 deep. cAdvisor's 30s housekeeping is the suspected repeat caller, unconfirmed (logs were empty). During the pile-up even `docker images` times out; lightweight socket endpoints (`/info`, `/_ping`) stay fast — diagnose with those, don't add heavy CLI calls to the backlog. Mitigated since 2026-08-10 by the daily unused-image prune (dotfiles `docker-image-prune.timer`, `-a -f --filter until=24h`): 213→84 images, 108→37GB, and no recurrence on subsequent reboots. Remaining work: - [ ] confirm whether it still reproduces on cold boot at current image volume; if yes, pin the repeat caller (cAdvisor housekeeping interval is tunable) - [ ] consider upstream report: overlapping-call stacking in `ImageService.Images()` is arguably a moby bug (no request coalescing/backpressure on an expensive endpoint)

mysticalsoap added the

bug

label

2026-08-27 11:56:44 -04:00

mysticalsoap referenced this issue

2026-08-27 11:57:05 -04:00

Post-Podman: replace systemd units with Dagu? #120