monitoring: loki ruler and log-based alerts #393

Merged

mysticalsoap merged 3 commits from monitoring/loki-ruler into trunk

2026-10-07 22:28:48 -04:00

mysticalsoap commented

2026-10-07 22:05:37 -04:00

Owner

Problem

Promtail ships Traefik, Authelia and Jellyfin logs into Loki, but Loki had no ruler block and no Alertmanager URL, so the logs could only be queried by hand. Closes #105.

Fix

loki/config.yml: ruler.alertmanager_url, rules dir moved to /etc/loki/rules and bind-mounted from monitoring/loki/rules/ (directory mount, so edits are live within the ruler's 1m poll). The fake/ tenant dir is explained in the config comment.
monitoring/loki/rules/fake/alerts.yml, logs group: TraefikLogSilent (30m), AutheliaLogSilent (12h), JellyfinLogSilent (6h), AutheliaUnexpectedError. Windows and exclusions come from 7 days of Loki data; details in the commit body and Monitoring.md.
Prometheus now scrapes loki:3100 and promtail:9080 (TargetDown covers a dead ruler) and gets LokiRuleEvaluationFailing / LokiAlertmanagerNotifyFailing. There is no CI lint for the Loki rules file: the image ships no lokitool and promtool rejects LogQL.
Promtail labels container streams job=docker; docker_sd_configs sets no job label, so {job="docker"} matched nothing before.
Monitoring.md: architecture line, logs group table, the Loki query-log gotcha (a container-wide pattern rule matches its own query text in {container="loki"}).

Left out on purpose: a Traefik 5xx rule (status codes are already a metric label, belongs in alerts.yml if wanted) and any container-wide error-pattern rule (self-matching per the gotcha, and error-ish lines run to hundreds a day for several containers).

New mount is git-tracked, nothing for includes.sh.

Verification

just up monitoring, then just recreate prometheus and just recreate promtail (bind-mounted single files, compose doesn't recreate on content change).

/prometheus/api/v1/rules on Loki: group logs, all four rules health: ok, state: inactive; loki_ruler_managers_total 1, loki_prometheus_rule_evaluation_failures_total 0.
End-to-end: a temporary LokiRulerTest rule (absent_over_time({job="nonexistent"}[5m]) == 1) appended to the rules file was picked up by the ruler without a restart, fired, and showed up in Alertmanager's /api/v2/alerts with its labels and annotations intact; loki_prometheus_notifications_sent_total 1, errors_total 0. Pre-silenced in Alertmanager so it never reached ntfy; rule and silence removed afterwards.
Expressions checked directly against Loki: absent_over_time on a live job returns nothing, on a missing job returns 1. The Authelia rule returns nothing for the current 10m window; over the past 7 days it would have fired on five distinct messages (two OIDC client misconfigs, one LDAP startup-check failure and its fatal follow-up, one benign prompt=none case).
Prometheus /api/v1/targets: loki and promtail up; both new rules ok/inactive; promtool check rules passes (36 rules).
Loki /loki/api/v1/label/job/values now lists docker.

🤖 Generated with Claude Code

## Problem Promtail ships Traefik, Authelia and Jellyfin logs into Loki, but Loki had no `ruler` block and no Alertmanager URL, so the logs could only be queried by hand. Closes #105. ## Fix - `loki/config.yml`: `ruler.alertmanager_url`, rules dir moved to `/etc/loki/rules` and bind-mounted from `monitoring/loki/rules/` (directory mount, so edits are live within the ruler's 1m poll). The `fake/` tenant dir is explained in the config comment. - `monitoring/loki/rules/fake/alerts.yml`, `logs` group: `TraefikLogSilent` (30m), `AutheliaLogSilent` (12h), `JellyfinLogSilent` (6h), `AutheliaUnexpectedError`. Windows and exclusions come from 7 days of Loki data; details in the commit body and `Monitoring.md`. - Prometheus now scrapes `loki:3100` and `promtail:9080` (TargetDown covers a dead ruler) and gets `LokiRuleEvaluationFailing` / `LokiAlertmanagerNotifyFailing`. There is no CI lint for the Loki rules file: the image ships no `lokitool` and promtool rejects LogQL. - Promtail labels container streams `job=docker`; `docker_sd_configs` sets no `job` label, so `{job="docker"}` matched nothing before. - `Monitoring.md`: architecture line, `logs` group table, the Loki query-log gotcha (a container-wide pattern rule matches its own query text in `{container="loki"}`). Left out on purpose: a Traefik 5xx rule (status codes are already a metric label, belongs in `alerts.yml` if wanted) and any container-wide error-pattern rule (self-matching per the gotcha, and `error`-ish lines run to hundreds a day for several containers). New mount is git-tracked, nothing for `includes.sh`. ## Verification `just up monitoring`, then `just recreate prometheus` and `just recreate promtail` (bind-mounted single files, compose doesn't recreate on content change). - `/prometheus/api/v1/rules` on Loki: group `logs`, all four rules `health: ok`, `state: inactive`; `loki_ruler_managers_total 1`, `loki_prometheus_rule_evaluation_failures_total 0`. - End-to-end: a temporary `LokiRulerTest` rule (`absent_over_time({job="nonexistent"}[5m]) == 1`) appended to the rules file was picked up by the ruler without a restart, fired, and showed up in Alertmanager's `/api/v2/alerts` with its labels and annotations intact; `loki_prometheus_notifications_sent_total 1`, `errors_total 0`. Pre-silenced in Alertmanager so it never reached ntfy; rule and silence removed afterwards. - Expressions checked directly against Loki: `absent_over_time` on a live job returns nothing, on a missing job returns 1. The Authelia rule returns nothing for the current 10m window; over the past 7 days it would have fired on five distinct messages (two OIDC client misconfigs, one LDAP startup-check failure and its fatal follow-up, one benign `prompt=none` case). - Prometheus `/api/v1/targets`: `loki` and `promtail` up; both new rules `ok`/`inactive`; `promtool check rules` passes (36 rules). - Loki `/loki/api/v1/label/job/values` now lists `docker`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added 3 commits

2026-10-07 22:05:37 -04:00

monitoring: enable the loki ruler with a first rule set c0734b123b

Loki's ruler evaluates LogQL alert rules and posts to Alertmanager the
same way Prometheus does, so log-based alerting is one config block, a
rules directory and a mount. The rules sit next to config.yml rather
than under the data mount: the data dir stays opaque, and a directory
bind means an edit is live within the ruler's 1m poll instead of
needing the restart a single-file bind does.

The first rules are what metrics can't express. A log source going
silent is a pipeline fault (Promtail, logrotate's reopen signal, or
the app) that also blinds CrowdSec, which tails the same three files.
Authelia's startup-check failures and OIDC client misconfigs have no
metric at all. Windows come from 7 days of Loki data: Traefik never
dropped under ~380 lines per 10m, Authelia under ~25 per 6h, Jellyfin
under ~85 per 2h. The Authelia exclusions are the messages that
dominated the error level over that week and are either covered by
AutheliaHighFailedAuthn or caused by scanners.

No container-wide pattern rule: Loki logs every query it runs,
including the ruler's, so such a rule matches its own text in Loki's
stream on every evaluation. No Traefik 5xx rule either, since the
status code is already a metric label.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

monitoring: scrape loki and promtail, alert on ruler failures 24c40fb69c

Both expose /metrics and nothing scraped them, so a dead ruler would
have taken the log rules with it silently. TargetDown now covers that.

Nothing lints the Loki rules file before it loads: the Loki image
ships no lokitool, and promtool rejects LogQL. A rule that parses but
cannot evaluate, or a ruler that cannot reach Alertmanager, is only
visible in the ruler's own counters, so two rules watch those.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

promtail: label container streams job=docker

validate / validate (pull_request) Successful in 24s

Details

6e9e20ff4d

docker_sd_configs targets carry no job label, unlike static_configs,
so {job="docker"} matched nothing and the container streams were only
reachable through the compose labels. Chunks already written keep
their old label set and age out with the 30d retention; no dashboard
queries Loki yet, so nothing depends on either shape.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap merged commit 067cd7c66f into trunk

2026-10-07 22:28:48 -04:00

mysticalsoap deleted branch monitoring/loki-ruler

2026-10-07 22:28:48 -04:00

mysticalsoap referenced this pull request from a commit

2026-10-07 22:28:49 -04:00

Merge pull request 'monitoring: loki ruler and log-based alerts' (#393) from monitoring/loki-ruler into trunk

mysticalsoap referenced this pull request

2026-10-08 12:50:09 -04:00

Couple of alert issues #404