monitoring: readable ntfy notifications #395

Merged

mysticalsoap merged 2 commits from monitoring/ntfy-readability into trunk

2026-10-07 23:59:50 -04:00

mysticalsoap commented

2026-10-07 23:35:58 -04:00

Owner

Problem

ntfy notifications are hard to read: the title is [FIRING:1] Alertname and the body is every label and annotation of every alert in the group. For cAdvisor-fed alerts that is 21 container_label_com_docker_compose_* labels on top of the useful ones.

Fix

Two commits, independent:

cadvisor: --store_container_labels=false plus a whitelist of the three compose labels the cadvisor dashboard reads (project, service, project.working_dir). A container series goes from 27 labels to 6. Nothing else in the repo queries a container_label_*.
ntfy-alertmanager: template-path pointing at a tracked body.tmpl (Go text/template, no extra functions): per alert, the summary annotation, then description if set. Every rule already has a summary. Plain text, no markdown, since the Android app's markdown rendering is undocumented. The title is built by the bridge and cannot be templated.

Docs: Monitoring.md (template, label whitelist), Services.md row.

Verification

just recreate ntfy-alertmanager-config re-rendered the config; grep -c template-path in the running bridge's config gives 1. just up monitoring recreated cadvisor and ntfy-alertmanager, both healthy.
cAdvisor /metrics for name="jellyfin": labels are now container_label_com_docker_compose_project, ..._service, ..._project_working_dir, id, image, name.
amtool alert add alertname=NotificationTemplateTest severity=warning category=containers with a summary and description annotation, 4-minute end: Alertmanager's webhook counter went 5 → 6 with every _failed_total reason at 0, ntfy's messages_published advanced, and the bridge logged no template error. Rendered body confirmed in the Android app: two lines, summary then description, no label list.

🤖 Generated with Claude Code

## Problem ntfy notifications are hard to read: the title is `[FIRING:1] Alertname` and the body is every label and annotation of every alert in the group. For cAdvisor-fed alerts that is 21 `container_label_com_docker_compose_*` labels on top of the useful ones. ## Fix Two commits, independent: 1. **cadvisor**: `--store_container_labels=false` plus a whitelist of the three compose labels the cadvisor dashboard reads (`project`, `service`, `project.working_dir`). A container series goes from 27 labels to 6. Nothing else in the repo queries a `container_label_*`. 2. **ntfy-alertmanager**: `template-path` pointing at a tracked `body.tmpl` (Go `text/template`, no extra functions): per alert, the `summary` annotation, then `description` if set. Every rule already has a summary. Plain text, no markdown, since the Android app's markdown rendering is undocumented. The title is built by the bridge and cannot be templated. Docs: Monitoring.md (template, label whitelist), Services.md row. ## Verification - `just recreate ntfy-alertmanager-config` re-rendered the config; `grep -c template-path` in the running bridge's config gives 1. `just up monitoring` recreated cadvisor and ntfy-alertmanager, both healthy. - cAdvisor `/metrics` for `name="jellyfin"`: labels are now `container_label_com_docker_compose_project`, `..._service`, `..._project_working_dir`, `id`, `image`, `name`. - `amtool alert add alertname=NotificationTemplateTest severity=warning category=containers` with a summary and description annotation, 4-minute end: Alertmanager's webhook counter went 5 → 6 with every `_failed_total` reason at 0, ntfy's `messages_published` advanced, and the bridge logged no template error. Rendered body confirmed in the Android app: two lines, summary then description, no label list. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added 2 commits

2026-10-07 23:35:58 -04:00

cadvisor: export only the compose labels the dashboard reads a7154236f9

store_container_labels copies every container label onto every series:
21 of a container's 27 labels were container_label_com_docker_compose_*,
and each alert carried all of them into ntfy. The whitelist keeps the
three the cadvisor dashboard queries; nothing else in the repo reads any.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

monitoring: render ntfy bodies from a template

validate / validate (pull_request) Successful in 24s

Details

5887f6ab4e

The bridge's default body lists every label and annotation of every alert
in the group. body.tmpl prints each alert's summary and description
instead; every rule already has a summary. Plain text rather than
markdown: the Android app's markdown support is undocumented, and the
summary reads fine without it. The title stays the bridge's own
"[FIRING:n] Alertname"; template-path only covers the body.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap merged commit 3c3010fd70 into trunk

2026-10-07 23:59:50 -04:00

mysticalsoap deleted branch monitoring/ntfy-readability

2026-10-07 23:59:50 -04:00

mysticalsoap referenced this pull request from a commit

2026-10-07 23:59:50 -04:00

Merge pull request 'monitoring: readable ntfy notifications' (#395) from monitoring/ntfy-readability into trunk

mysticalsoap referenced this pull request

2026-10-08 00:10:27 -04:00

monitoring: drop the ntfy bridge, template in ntfy #396