dagu: weekly trivy scan of running images, with alerts

mysticalsoap commented

2026-09-19 22:12:23 -04:00

Owner

Problem

Digest pinning is a deliberate opt-out of silent security patches, and Renovate only ever says "a newer digest exists", never "the one you run is CVE'd". Nothing ranks its PRs, and the locally built images (dagu-tools, config-renderer, knock, metadata) are rebuilt by nobody, so they can sit on stale base packages for months without anything noticing.

Closes #108. Deviations from the issue as filed: Trivy, not Grype; and the framing is narrower than written — Renovate does open a digest PR whenever upstream rebuilds, so the scan is the priority signal plus coverage of the local builds, not the only CVE signal.

Fix

dagu-scanner sidecar (Trivy image plus curl/jq) and an image-scan DAG, Monday 06:00. scan.sh lists running containers through the socket proxy and scans each distinct image by id via the daemon, so a re-pulled tag or a rebuilt local image can't mask what actually runs. Fix-available HIGH/CRITICAL only, .trivyignore for accepted CVEs, results to image_scan.prom beside the backup/SMART metrics.
Socket proxy: three read-only GET routes (container list, image inspect/history, image export) and HEAD /_ping. Nothing mutating.
Daemon-first, registry fallback: two images come back from docker save as a 5-blob tar against a 29-layer manifest under the containerd image store (cause not pinned down; no matching Trivy issue upstream). Those are pulled from their registry by the same digest instead.
images alert group: ImageVulnerableCritical / ImageVulnerableHigh per image, ImageScanFailed, ImageScanStale (14d), ImageScanMetricMissing. Category images, so they land on homelab-alerts like backup and smart. A 7d repeat_interval route for the category, and Alertmanager --data.retention=192h so it doesn't forget it already notified.
Docs: Maintenance (job), Monitoring (textfile table, alert group), Networking (proxy callers), README tree.

Verification

just up dagu, then docker exec dagu dagu start image-scan three times on the branch. Cold (cache cleared): 58 images in 145s, scanner peak 354MiB of the 1g limit. Warm: 10s. Proxy log shows no blocked requests during a run.
Before the registry fallback, rreading-glasses and invidious-companion failed with not found in tar; after it, 58/58 image_scan_image_ok 1.
promtool check rules (35 rules) and amtool check-config clean. Prometheus and Alertmanager restarted on the branch: ImageVulnerableCritical firing on 14 images, ImageVulnerableHigh on 43, ImageScanFailed cleared once the fallback landed. ntfy's messages_published counter rose by 3 in the window, one per grouped alert.
Alertmanager recreated with the retention flag: the repeat_interval is greater than the data retention period warning is gone.

Review notes and follow-ups

First-run reality: 14 images carry a fixable CRITICAL, 43 a fixable HIGH (grafana and seerr at 104 HIGH each). The HIGH rule will be a permanently firing weekly wall of text. Suggest dropping it and leaving HIGH to the metric and Grafana — one-line change either way, your call.
_helper/knock/Dockerfile and _helper/config-renderer/Dockerfile have unpinned FROM lines, against the pinning policy. Separate PR.
just deploy never rebuilds build: images, so a Renovate bump of any FROM line (this PR's dagu/scanner/Dockerfile included) changes nothing live until a manual build. Worth an issue.

## Problem Digest pinning is a deliberate opt-out of silent security patches, and Renovate only ever says "a newer digest exists", never "the one you run is CVE'd". Nothing ranks its PRs, and the locally built images (dagu-tools, config-renderer, knock, metadata) are rebuilt by nobody, so they can sit on stale base packages for months without anything noticing. Closes #108. Deviations from the issue as filed: Trivy, not Grype; and the framing is narrower than written — Renovate does open a digest PR whenever upstream rebuilds, so the scan is the priority signal plus coverage of the local builds, not the only CVE signal. ## Fix - `dagu-scanner` sidecar (Trivy image plus curl/jq) and an `image-scan` DAG, Monday 06:00. `scan.sh` lists running containers through the socket proxy and scans each distinct image **by id** via the daemon, so a re-pulled tag or a rebuilt local image can't mask what actually runs. Fix-available HIGH/CRITICAL only, `.trivyignore` for accepted CVEs, results to `image_scan.prom` beside the backup/SMART metrics. - Socket proxy: three read-only GET routes (container list, image inspect/history, image export) and `HEAD /_ping`. Nothing mutating. - Daemon-first, registry fallback: two images come back from `docker save` as a 5-blob tar against a 29-layer manifest under the containerd image store (cause not pinned down; no matching Trivy issue upstream). Those are pulled from their registry by the same digest instead. - `images` alert group: `ImageVulnerableCritical` / `ImageVulnerableHigh` per image, `ImageScanFailed`, `ImageScanStale` (14d), `ImageScanMetricMissing`. Category `images`, so they land on `homelab-alerts` like backup and smart. A 7d `repeat_interval` route for the category, and Alertmanager `--data.retention=192h` so it doesn't forget it already notified. - Docs: Maintenance (job), Monitoring (textfile table, alert group), Networking (proxy callers), README tree. ## Verification - `just up dagu`, then `docker exec dagu dagu start image-scan` three times on the branch. Cold (cache cleared): 58 images in 145s, scanner peak 354MiB of the 1g limit. Warm: 10s. Proxy log shows no blocked requests during a run. - Before the registry fallback, rreading-glasses and invidious-companion failed with `not found in tar`; after it, 58/58 `image_scan_image_ok 1`. - `promtool check rules` (35 rules) and `amtool check-config` clean. Prometheus and Alertmanager restarted on the branch: `ImageVulnerableCritical` firing on 14 images, `ImageVulnerableHigh` on 43, `ImageScanFailed` cleared once the fallback landed. ntfy's `messages_published` counter rose by 3 in the window, one per grouped alert. - Alertmanager recreated with the retention flag: the `repeat_interval is greater than the data retention period` warning is gone. ## Review notes and follow-ups - First-run reality: 14 images carry a fixable CRITICAL, 43 a fixable HIGH (grafana and seerr at 104 HIGH each). The HIGH rule will be a permanently firing weekly wall of text. Suggest dropping it and leaving HIGH to the metric and Grafana — one-line change either way, your call. - `_helper/knock/Dockerfile` and `_helper/config-renderer/Dockerfile` have unpinned `FROM` lines, against the pinning policy. Separate PR. - `just deploy` never rebuilds `build:` images, so a Renovate bump of any `FROM` line (this PR's `dagu/scanner/Dockerfile` included) changes nothing live until a manual build. Worth an issue.

mysticalsoap added 2 commits

2026-09-19 22:12:23 -04:00

dagu: weekly trivy scan of running images 528af868d8

Digest pinning opts out of silent security patches, and Renovate only says a
newer digest exists. The scan says which running digests carry a fixable
HIGH/CRITICAL CVE, and covers the locally built images nothing else rebuilds.

Through the daemon rather than the registries: it sees exactly the running
set including local builds, downloads nothing, and stays clear of Docker
Hub's 100-pulls-per-6h anonymous quota that Renovate already draws on.
Trivy's daemon source makes three read-only GETs (inspect, history, export),
now on the socket proxy allowlist; export is lazy, only for layers missing
from the cache volume.

Two images (rreading-glasses, invidious-companion) come back from the daemon
as a 5-blob tar for a 29-layer manifest under the containerd image store.
Cause not pinned down and upstream has no matching issue, so those fall back
to their registry by the same digest.

mem_limit 1g: a cold run over 58 images peaks around 350MiB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

monitoring: alert on fixable image vulnerabilities