Couple of alert issues #404

New issue

Closed

opened 2026-10-08 12:15:03 -04:00 by mysticalsoap · 1 comment

mysticalsoap commented

2026-10-08 12:15:03 -04:00

Owner

After a recent system upgrade and restart got a couple of alerts that need addressing/refining:

qbit_manage has not completed a pass in over 13h (did resolve itself)
Alertmanager failed to deliver webhook notifications (also resolved itself, also has small issue of saying "1 failures", not sure if easily fixable grammar)
a few unexpected Authelia errors that resolved themselves but should be looked at. These errors also stack up to make them a lot less readable so readability should also be looked at?

After a recent system upgrade and restart got a couple of alerts that need addressing/refining: 1. qbit_manage has not completed a pass in over 13h (did resolve itself) 2. Alertmanager failed to deliver webhook notifications (also resolved itself, also has small issue of saying "1 failures", not sure if easily fixable grammar) 3. a few unexpected Authelia errors that resolved themselves but should be looked at. These errors also stack up to make them a lot less readable so readability should also be looked at?

mysticalsoap added the

general-admin

label

2026-10-08 12:15:03 -04:00

mysticalsoap commented

2026-10-08 12:50:09 -04:00

Author

Owner

Findings from the logs and Prometheus/Loki history, all times local.

1. QbitManageStale — two scheduled passes skipped, cause was just run-job

Last success before the alert: 19:46 on 10-07, the just run-job qbit-manage from #388's verification. docker exec runs as root (the entrypoint's PUID drop doesn't apply to exec), so that run wrote its dag-run, proc and event files as root. The scheduler runs as 948 and from 19:46:40 logged permission denied on them every 30s (~200/h).
00:00 and 06:00 on 10-08: Failed to check if DAG is running ... proc/qbit-manage: permission denied, run never dispatched. 13h after 19:46 is 08:46 → alert at 08:50.
The reboot fixed it: dagu-data-init chowns the volume on every start. 12:00 pass ran and marked.
Fix: docker exec --user "$(id -u apps):$(id -g apps)" in the run-job recipe. Verified a 948 exec can read the password file (owned 948, 600) and run dagu status. Every ad-hoc run (just prune, just restart-invidious) has had this effect since the recipe started working on 10-07.

2. AlertmanagerNotifyFailing — ntfy's template bind mount went stale at the #396 sync

ntfy was recreated at 00:07 (branch deploy). The checkout to trunk at 00:27:18 deleted monitoring/ntfy/templates/ (trunk didn't have it yet); the pull at 00:27:19 recreated it as a new inode. The container's bind mount stayed on the deleted directory, so every templated webhook got 400 template file not found.
First real alert after that was QbitManageStale at 08:50; every notification failed until the reboot re-bound the mount (16 failures 08:50–11:45). The AlertmanagerNotifyFailing page itself was blocked the same way, which is why it only arrived after boot. just deploy --dry-run can't see this.
Rule for the sync procedure: a branch that adds a bind-mounted file or directory needs just recreate <container> after the checkout+pull. Goes in CONTRIBUTING (live tree rule) and the Monitoring gotchas.
"1 failures": {{ $value }} failures with no pluralisation. Trivial template conditional. The count is also meaningless across an Alertmanager restart (counter reset), so wording it as a count is misleading anyway; "N failed deliveries in 15m" with the singular handled is enough.

3. AutheliaUnexpectedError — three messages, one group

msg	when	verdict
`Error occurred running a startup check` (provider=user, LDAP connection refused)	11:48:46, once	Boot race: dockerd restarts containers without `depends_on` ordering, Authelia came up before lldap, exited, restart policy brought it back. Benign once; a real LDAP problem repeats every restart.
`One or more providers had fatal failures performing startup checks`	11:48:46, once	Same event, second line.
`Authorization Request ... client 'forgejo' ... 'prompt' type of 'none' was requested but the user is not logged in`	11:50:46	Forgejo's silent-login probe after the session was gone; `login_required` is a normal OIDC outcome, Authelia just logs it at error. Predicted in #393.

Proposed:

Exclude the prompt' type of 'none' message outright.
Keep the startup-check pair but require > 1 in 10m (a persistent failure logs on every restart, a boot race logs once). Either a second rule or a threshold on msg =~ ".*startup check.*".
Readability: the group body repeats Authelia logged an unexpected error three times, once per alert, each followed by the full msg. Template change: when all alerts in a group share a summary, title it <summary> (3) and list only the descriptions. And trim the msg label in the Loki rule (label_format with trunc) so a request id and client name don't make one line 200 chars.

Aside, not alerted on: ~160 Unsuccessful 1FA authentication attempt by user '' today in bursts of ~7 roughly hourly at :43–:44 — one client retrying with empty credentials. Below AutheliaHighFailedAuthn's threshold by design; worth identifying separately.

Findings from the logs and Prometheus/Loki history, all times local. **1. QbitManageStale — two scheduled passes skipped, cause was `just run-job`** - Last success before the alert: 19:46 on 10-07, the `just run-job qbit-manage` from #388's verification. `docker exec` runs as root (the entrypoint's PUID drop doesn't apply to exec), so that run wrote its dag-run, proc and event files as root. The scheduler runs as 948 and from 19:46:40 logged `permission denied` on them every 30s (~200/h). - 00:00 and 06:00 on 10-08: `Failed to check if DAG is running ... proc/qbit-manage: permission denied`, run never dispatched. 13h after 19:46 is 08:46 → alert at 08:50. - The reboot fixed it: `dagu-data-init` chowns the volume on every start. 12:00 pass ran and marked. - Fix: `docker exec --user "$(id -u apps):$(id -g apps)"` in the `run-job` recipe. Verified a 948 exec can read the password file (owned 948, 600) and run `dagu status`. Every ad-hoc run (`just prune`, `just restart-invidious`) has had this effect since the recipe started working on 10-07. **2. AlertmanagerNotifyFailing — ntfy's template bind mount went stale at the #396 sync** - ntfy was recreated at 00:07 (branch deploy). The checkout to trunk at 00:27:18 deleted `monitoring/ntfy/templates/` (trunk didn't have it yet); the pull at 00:27:19 recreated it as a new inode. The container's bind mount stayed on the deleted directory, so every templated webhook got `400 template file not found`. - First real alert after that was QbitManageStale at 08:50; every notification failed until the reboot re-bound the mount (16 failures 08:50–11:45). The AlertmanagerNotifyFailing page itself was blocked the same way, which is why it only arrived after boot. `just deploy --dry-run` can't see this. - Rule for the sync procedure: a branch that adds a bind-mounted file or directory needs `just recreate <container>` after the checkout+pull. Goes in CONTRIBUTING (live tree rule) and the Monitoring gotchas. - "1 failures": `{{ $value }} failures` with no pluralisation. Trivial template conditional. The count is also meaningless across an Alertmanager restart (counter reset), so wording it as a count is misleading anyway; "N failed deliveries in 15m" with the singular handled is enough. **3. AutheliaUnexpectedError — three messages, one group** | msg | when | verdict | |---|---|---| | `Error occurred running a startup check` (provider=user, LDAP connection refused) | 11:48:46, once | Boot race: dockerd restarts containers without `depends_on` ordering, Authelia came up before lldap, exited, restart policy brought it back. Benign once; a real LDAP problem repeats every restart. | | `One or more providers had fatal failures performing startup checks` | 11:48:46, once | Same event, second line. | | `Authorization Request ... client 'forgejo' ... 'prompt' type of 'none' was requested but the user is not logged in` | 11:50:46 | Forgejo's silent-login probe after the session was gone; `login_required` is a normal OIDC outcome, Authelia just logs it at error. Predicted in #393. | Proposed: - Exclude the `prompt' type of 'none'` message outright. - Keep the startup-check pair but require `> 1` in 10m (a persistent failure logs on every restart, a boot race logs once). Either a second rule or a threshold on `msg =~ ".*startup check.*"`. - Readability: the group body repeats `Authelia logged an unexpected error` three times, once per alert, each followed by the full msg. Template change: when all alerts in a group share a summary, title it `<summary> (3)` and list only the descriptions. And trim the msg label in the Loki rule (`label_format` with `trunc`) so a request id and client name don't make one line 200 chars. Aside, not alerted on: ~160 `Unsuccessful 1FA authentication attempt by user ''` today in bursts of ~7 roughly hourly at :43–:44 — one client retrying with empty credentials. Below AutheliaHighFailedAuthn's threshold by design; worth identifying separately.

mysticalsoap referenced this issue from a pull request that will close it,

2026-10-08 13:04:55 -04:00

alerts: #404 findings -- run-job uid, ntfy group layout, Authelia noise #406

mysticalsoap closed this issue

2026-10-08 13:40:25 -04:00

mysticalsoap referenced this issue from a commit

2026-10-08 13:40:27 -04:00

Merge pull request 'alerts: #404 findings -- run-job uid, ntfy group layout, Authelia noise' (#406) from alerts/404-findings into trunk

mysticalsoap referenced this issue

2026-10-08 14:00:07 -04:00

authelia: forward-auth accepts the session cookie only #409