monitoring: btrfs scrub for / and /mnt/support, schedules page #405

Merged

mysticalsoap merged 2 commits from btrfs-scrub into trunk

2026-10-08 12:28:43 -04:00

mysticalsoap commented

2026-10-08 12:24:38 -04:00

Owner

Closes #124.

Problem

Only /mnt/external had a scrub timer. The LUKS root (every app database, the restic source data) and /mnt/support (the restic repo) were never checksummed beyond what a read happened to touch, and a scrub that found errors would have reported them nowhere but the journal.

Separately, timers are spread over system units, user units, Dagu DAGs and distro packages, so picking a slot for a new job meant reading all four.

Fix

Drop-ins on the distro's btrfs-scrub@ templates under _helper/monitoring/systemd/: the service drop-in makes every instance oneshot with an ExecStartPost that writes btrfs_scrub_last_success_timestamp_seconds (skipped on a non-zero exit, so uncorrectable errors leave the timestamp stale); per-instance timer drop-ins pin / to the 1st and /mnt/support to the 2nd at 14:00, clear of the 03:15 backup.
New btrfs alert group: BtrfsDeviceErrors on node-exporter's existing node_btrfs_device_errors_total (the kernel's btrfs device stats, which a scrub bumps on a bad checksum), BtrfsScrubStale at 35d, BtrfsScrubMetricMissing. /mnt/external is excluded from alerting; it sits unplugged for months.
Schedules.md: every timer on the host in one table by clock time, with what each restarts or reads heavily. Linked from the docs index and Maintenance.
Maintenance and Monitoring sections for the above. The dotfiles README now points here for scrub (mysticalsoap/dotfiles@c06da0f).

Verification

promtool check rules passes; Prometheus restarted on the branch's rules, all three evaluate with health: ok.
Drop-ins installed and timers enabled on the host: next runs 2026-11-01 14:00 (/) and 2026-11-02 14:00 (/mnt/support).
Both scrubs run once by hand: / 345.72 GiB in 2:50 at 2.03 GiB/s, /mnt/support 8.39 GiB in 2 s, no errors on either, device stats all zero.
The first run exposed that upstream's default Type=simple fires ExecStartPost the moment the scrub starts; with Type=oneshot the mark script's PID and the metric file's mtime both follow the scrub's exit.
btrfs_scrub_last_success_timestamp_seconds present in Prometheus for both mounts; BtrfsScrubMetricMissing went pending → inactive once they appeared.

🤖 Generated with Claude Code

Closes #124. ## Problem Only `/mnt/external` had a scrub timer. The LUKS root (every app database, the restic source data) and `/mnt/support` (the restic repo) were never checksummed beyond what a read happened to touch, and a scrub that found errors would have reported them nowhere but the journal. Separately, timers are spread over system units, user units, Dagu DAGs and distro packages, so picking a slot for a new job meant reading all four. ## Fix - Drop-ins on the distro's `btrfs-scrub@` templates under `_helper/monitoring/systemd/`: the service drop-in makes every instance `oneshot` with an `ExecStartPost` that writes `btrfs_scrub_last_success_timestamp_seconds` (skipped on a non-zero exit, so uncorrectable errors leave the timestamp stale); per-instance timer drop-ins pin `/` to the 1st and `/mnt/support` to the 2nd at 14:00, clear of the 03:15 backup. - New `btrfs` alert group: `BtrfsDeviceErrors` on node-exporter's existing `node_btrfs_device_errors_total` (the kernel's `btrfs device stats`, which a scrub bumps on a bad checksum), `BtrfsScrubStale` at 35d, `BtrfsScrubMetricMissing`. `/mnt/external` is excluded from alerting; it sits unplugged for months. - `Schedules.md`: every timer on the host in one table by clock time, with what each restarts or reads heavily. Linked from the docs index and Maintenance. - Maintenance and Monitoring sections for the above. The dotfiles README now points here for scrub (`mysticalsoap/dotfiles@c06da0f`). ## Verification - `promtool check rules` passes; Prometheus restarted on the branch's rules, all three evaluate with `health: ok`. - Drop-ins installed and timers enabled on the host: next runs 2026-11-01 14:00 (`/`) and 2026-11-02 14:00 (`/mnt/support`). - Both scrubs run once by hand: `/` 345.72 GiB in 2:50 at 2.03 GiB/s, `/mnt/support` 8.39 GiB in 2 s, no errors on either, device stats all zero. - The first run exposed that upstream's default `Type=simple` fires `ExecStartPost` the moment the scrub starts; with `Type=oneshot` the mark script's PID and the metric file's mtime both follow the scrub's exit. - `btrfs_scrub_last_success_timestamp_seconds` present in Prometheus for both mounts; `BtrfsScrubMetricMissing` went pending → inactive once they appeared. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added 2 commits

2026-10-08 12:24:38 -04:00

docs: schedules page listing every timer 3776bcebfa

Timers are spread over system units, user units, Dagu DAGs and distro
packages, so checking what a new job would collide with meant reading
all four. One table ordered by clock time, with what each job restarts
or reads heavily, since restart-type jobs are the ones that bite.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

monitoring: scrub / and /mnt/support monthly, alert on btrfs errors

validate / validate (pull_request) Successful in 19s

Details

4d85c9768c

Only /mnt/external had a scrub timer, so the LUKS root holding every
app database and /mnt/support holding the restic repo were never
checksummed beyond what a read happened to touch.

node-exporter's btrfs collector already exports the kernel's device
error counters, and a scrub that finds a bad checksum bumps them, so
the error alert needs no new metric. What did need one is whether a
scrub ran at all: a drop-in on the distro's service template covers
every instance with one file, and ExecStartPost is skipped on a
non-zero exit, which gives success-only semantics without a wrapper.

Fixed 14:00 slots replace upstream's monthly plus up to a week of
random delay: clear of the 03:15 backup, and tight enough for a 35-day
staleness alert. /mnt/external is excluded from alerting because it
sits unplugged for months.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap force-pushed btrfs-scrub from 4d85c9768c

validate / validate (pull_request) Successful in 19s

Details

to 4acfeb965b

validate / validate (pull_request) Successful in 18s

Details

2026-10-08 12:26:59 -04:00

Compare

mysticalsoap merged commit 33f3716bda into trunk

2026-10-08 12:28:43 -04:00

mysticalsoap deleted branch btrfs-scrub

2026-10-08 12:28:43 -04:00