alerts: #404 findings -- run-job uid, ntfy group layout, Authelia noise #406

Merged

mysticalsoap merged 4 commits from alerts/404-findings into trunk

2026-10-08 13:40:25 -04:00

mysticalsoap commented

2026-10-08 13:04:55 -04:00

Owner

Problem

The three alerts in #404, each with a cause found in the logs (see the issue comment):

just run-job execs into Dagu as root, so a manual run leaves root-owned run files the uid-948 scheduler cannot read; the next scheduled runs of that DAG are skipped until the volume is re-chowned. QbitManageStale was two skipped passes after #388's verification run.
After the #396 sync, ntfy's templates bind mount pointed at the directory the trunk checkout had deleted, and every webhook failed with "template file not found" until the reboot. The dry run cannot see this. The failure alert also said "1 failures".
AutheliaUnexpectedError pages on every boot (one failed startup check while LDAP is still starting) and on Forgejo's prompt=none probe, and a group of three alerts repeated its summary three times.

Fix

run-job execs as the apps uid.
CONTRIBUTING: a branch that added a bind-mounted path needs just recreate <container> after the sync.
ntfy template: a group lists one bullet per alert, summary omitted when every alert shares it. AlertmanagerNotifyFailing wording handles the singular.
Loki: prompt=none and the startup-check pair excluded from AutheliaUnexpectedError; new AutheliaStartupCheckFailing (critical) on a repeat within 10m, per provider. The msg label gets OIDC request ids stripped and is capped at 160 chars.

Verification

Live, 2026-10-08:

just run-job qbit-manage through the new recipe: Result: Succeeded; the new dag-run directory is 948 948; zero permission denied lines in Dagu's log afterwards (they ran ~200/h before the reboot).
promtool check rules: 43 rules OK. Prometheus recreated; /api/v1/rules shows the new description.
Loki /prometheus/api/v1/rules lists AutheliaUnexpectedError and AutheliaStartupCheckFailing, both healthy. Both expressions evaluated against today's logs: zero series (today's three messages are now excluded, the startup check appeared once). The label template tested on today's Forgejo line: 160 chars, id stripped, message intact up to "the user is".

Template rendered by a throwaway ntfy serve of the same image against three Alertmanager payloads:

single alert, unchanged:

⚠️ qbit_manage has not completed a pass in over 13h
Last success 16h ago -- check the qbit-manage DAG in Dagu.

three alerts sharing a summary:

⚠️ Authelia logged an unexpected error (3 alerts)
• Error occurred running a startup check (1 in 10m)
• One or more providers had fatal failures performing startup checks (1 in 10m)
• Authorization Request on client with id 'forgejo' ... (1 in 10m)

mixed summaries, group resolved with one still firing:

✅ Container a is failing its healthcheck (2 alerts)
• Container a is failing its healthcheck
  Unhealthy by its own healthcheck for 6m; not restarted automatically.
• [firing] Container b was OOM-killed

Closes #404

🤖 Generated with Claude Code

## Problem The three alerts in #404, each with a cause found in the logs (see the issue comment): - `just run-job` execs into Dagu as root, so a manual run leaves root-owned run files the uid-948 scheduler cannot read; the next scheduled runs of that DAG are skipped until the volume is re-chowned. QbitManageStale was two skipped passes after #388's verification run. - After the #396 sync, ntfy's templates bind mount pointed at the directory the trunk checkout had deleted, and every webhook failed with "template file not found" until the reboot. The dry run cannot see this. The failure alert also said "1 failures". - AutheliaUnexpectedError pages on every boot (one failed startup check while LDAP is still starting) and on Forgejo's `prompt=none` probe, and a group of three alerts repeated its summary three times. ## Fix - `run-job` execs as the apps uid. - CONTRIBUTING: a branch that added a bind-mounted path needs `just recreate <container>` after the sync. - ntfy template: a group lists one bullet per alert, summary omitted when every alert shares it. AlertmanagerNotifyFailing wording handles the singular. - Loki: `prompt=none` and the startup-check pair excluded from AutheliaUnexpectedError; new AutheliaStartupCheckFailing (critical) on a repeat within 10m, per provider. The `msg` label gets OIDC request ids stripped and is capped at 160 chars. ## Verification Live, 2026-10-08: - `just run-job qbit-manage` through the new recipe: `Result: Succeeded`; the new dag-run directory is `948 948`; zero `permission denied` lines in Dagu's log afterwards (they ran ~200/h before the reboot). - `promtool check rules`: 43 rules OK. Prometheus recreated; `/api/v1/rules` shows the new description. - Loki `/prometheus/api/v1/rules` lists `AutheliaUnexpectedError` and `AutheliaStartupCheckFailing`, both healthy. Both expressions evaluated against today's logs: zero series (today's three messages are now excluded, the startup check appeared once). The label template tested on today's Forgejo line: 160 chars, id stripped, message intact up to "the user is". - Template rendered by a throwaway `ntfy serve` of the same image against three Alertmanager payloads: single alert, unchanged: ``` ⚠️ qbit_manage has not completed a pass in over 13h Last success 16h ago -- check the qbit-manage DAG in Dagu. ``` three alerts sharing a summary: ``` ⚠️ Authelia logged an unexpected error (3 alerts) • Error occurred running a startup check (1 in 10m) • One or more providers had fatal failures performing startup checks (1 in 10m) • Authorization Request on client with id 'forgejo' ... (1 in 10m) ``` mixed summaries, group resolved with one still firing: ``` ✅ Container a is failing its healthcheck (2 alerts) • Container a is failing its healthcheck Unhealthy by its own healthcheck for 6m; not restarted automatically. • [firing] Container b was OOM-killed ``` Closes #404 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added 4 commits

2026-10-08 13:04:55 -04:00

justfile: run-job execs as the apps uid 91418e1165

docker exec runs as root: the image's entrypoint drops to PUID only
for the process it starts. A run started by the recipe wrote its
dag-run, proc and event files as root, and the scheduler (uid 948)
then got permission denied on every scan and skipped the next
scheduled runs of that DAG until dagu-data-init's chown at the next
recreate. The password file is owned by the same uid, so the CLI
still reads it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

monitoring: one bullet per alert in a group, plural in the delivery alert deb992787d

A group whose alerts all come from one rule (the per-msg Authelia
alerts) repeated the same summary before every description, three
blocks for three lines of content. The summary now appears once in
the title, and a group with mixed summaries keeps summary plus
description per bullet.

AlertmanagerNotifyFailing said "1 failures"; the count is also
unreliable across an Alertmanager restart, so the wording leans on the
window rather than the number.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: recreate a container whose bind-mounted path the branch added 3922e40598

A branch that adds a bind-mounted file or directory and is deployed
live, then synced with checkout trunk + pull, leaves the container
bound to the deleted inode: the checkout removes the path, the pull
recreates it. Every ntfy webhook got "template file not found" from
the #396 sync until the next reboot re-bound the mount, and the
notification-failure page was blocked by the same fault. The dry run
compares compose config, not inodes, so it stays empty.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

loki: hold the Authelia startup check to a repeat, drop two expected messages

validate / validate (pull_request) Successful in 19s

Details

ada0e2234f

Every boot logs one failed startup check: dockerd restarts containers
without depends_on order, Authelia beats LDAP up, exits, and the
restart policy brings it back. A provider that is really down logs it
on every restart, so a repeat within 10m is the signal and gets its
own critical rule with the provider in the description.

A client's prompt=none probe with nobody logged in is login_required,
a normal OIDC outcome Authelia logs at error level; Forgejo sends one
after every session loss.

The msg label is the alert text, so the OIDC request id is stripped
and the line capped; a 200-char message with an id in it was most of
a phone screen.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap referenced this pull request

2026-10-08 13:24:52 -04:00

boot: stop stacks through compose at shutdown so just up orders the start #407

mysticalsoap merged commit 4ad61d23d8 into trunk

2026-10-08 13:40:25 -04:00

mysticalsoap deleted branch alerts/404-findings

2026-10-08 13:40:25 -04:00

mysticalsoap referenced this pull request from a commit

2026-10-08 13:40:27 -04:00

Merge pull request 'alerts: #404 findings -- run-job uid, ntfy group layout, Authelia noise' (#406) from alerts/404-findings into trunk