monitoring: drop the ntfy bridge, template in ntfy #396

Merged

mysticalsoap merged 2 commits from monitoring/drop-ntfy-bridge into trunk

2026-10-08 00:26:48 -04:00

mysticalsoap commented

2026-10-08 00:10:27 -04:00

Owner

Problem

The ntfy-alertmanager bridge builds the notification title in code ([FIRING:n] Alertname), so #395's body template was as far as readability could go. The bridge also cost a container, a config-renderer one-shot, a bridge-password secret and an envsubst template.

Fix

Two commits.

Drop the bridge. ntfy renders Alertmanager's webhook JSON itself through monitoring/ntfy/templates/homelab.yml (template-dir, read on every request): title is the first alert's summary, prefixed 🚨 (critical) or ⚠️ (warning) while firing and ✅ once resolved, body is the description (or summary-plus-description per alert when a group holds several), priority follows severity. Alertmanager posts straight to http://ntfy:80/homelab-<topic>?template=homelab, one receiver per topic chosen by the existing category routes, with the ntfy token as a Bearer credentials_file. Removed: both bridge services, the alertmanager_bridge_password secret, config.scfg.template, body.tmpl, the backup include and the Secrets.md Pattern 2 exception. Tags can't be templated in ntfy, so none are set. Docs, README and topology regenerated.
Scrape Alertmanager and add AlertmanagerNotifyFailing: a bad template now fails only as a 400 on delivery, and nothing had a metric for that.

Verification

amtool check-config: 6 receivers, SUCCESS. promtool check config: 40 rules, SUCCESS. check-compose.sh ok for every stack.
up -d --wait --remove-orphans removed ntfy-alertmanager and ntfy-alertmanager-config, recreated ntfy (template mount) and alertmanager (secret, depends_on); all healthy. ntfy sees /etc/ntfy/templates/homelab.yml. Prometheus restarted for the edited bind mounts; up{job="alertmanager"} is 1 and the new rule is loaded.
Test alerts via amtool alert add: one critical single alert (category=containers) and a two-alert group (category=monitoring, so the fallback topic). Alertmanager's webhook counter went 0 → 2 with every _failed_total reason at 0, ntfy's messages_published advanced, no errors in either log. A render failure would have been a 400 and a clientError failure.
Rendered output is only visible in the app: homelab-containers should show a critical "Test: direct ntfy template, critical" titled message with the description as body; homelab-alerts a "Group test alpha (2 alerts)" message listing both. Resolved copies follow with a ✅ title and priority 1.

After merge, by hand: rm monitoring/secrets/alertmanager_bridge_password.txt (the secret file is outside what I touch).

🤖 Generated with Claude Code

## Problem The ntfy-alertmanager bridge builds the notification title in code (`[FIRING:n] Alertname`), so #395's body template was as far as readability could go. The bridge also cost a container, a config-renderer one-shot, a bridge-password secret and an envsubst template. ## Fix Two commits. 1. **Drop the bridge.** ntfy renders Alertmanager's webhook JSON itself through `monitoring/ntfy/templates/homelab.yml` (`template-dir`, read on every request): title is the first alert's `summary`, prefixed 🚨 (critical) or ⚠️ (warning) while firing and ✅ once resolved, body is the `description` (or summary-plus-description per alert when a group holds several), priority follows `severity`. Alertmanager posts straight to `http://ntfy:80/homelab-<topic>?template=homelab`, one receiver per topic chosen by the existing `category` routes, with the ntfy token as a Bearer `credentials_file`. Removed: both bridge services, the `alertmanager_bridge_password` secret, `config.scfg.template`, `body.tmpl`, the backup include and the Secrets.md Pattern 2 exception. Tags can't be templated in ntfy, so none are set. Docs, README and topology regenerated. 2. **Scrape Alertmanager** and add `AlertmanagerNotifyFailing`: a bad template now fails only as a 400 on delivery, and nothing had a metric for that. ## Verification - `amtool check-config`: 6 receivers, SUCCESS. `promtool check config`: 40 rules, SUCCESS. `check-compose.sh` ok for every stack. - `up -d --wait --remove-orphans` removed `ntfy-alertmanager` and `ntfy-alertmanager-config`, recreated `ntfy` (template mount) and `alertmanager` (secret, depends_on); all healthy. ntfy sees `/etc/ntfy/templates/homelab.yml`. Prometheus restarted for the edited bind mounts; `up{job="alertmanager"}` is 1 and the new rule is loaded. - Test alerts via `amtool alert add`: one critical single alert (`category=containers`) and a two-alert group (`category=monitoring`, so the fallback topic). Alertmanager's webhook counter went 0 → 2 with every `_failed_total` reason at 0, ntfy's `messages_published` advanced, no errors in either log. A render failure would have been a 400 and a `clientError` failure. - Rendered output is only visible in the app: `homelab-containers` should show a critical "Test: direct ntfy template, critical" titled message with the description as body; `homelab-alerts` a "Group test alpha (2 alerts)" message listing both. Resolved copies follow with a ✅ title and priority 1. After merge, by hand: `rm monitoring/secrets/alertmanager_bridge_password.txt` (the secret file is outside what I touch). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mysticalsoap added 2 commits

2026-10-08 00:10:27 -04:00

monitoring: drop the ntfy bridge, let ntfy template the webhooks 3f608f0763

ntfy renders a POSTed JSON body through Go templates for title, body and
priority (template-dir, since 2.14), so Alertmanager can post to it
directly. The bridge's title was fixed to "[FIRING:n] Alertname"; the
template puts the summary there instead.

Gone with the bridge: its config renderer, the envsubst template, the
bridge password secret, and one more image for Renovate. Topic routing
moves to Alertmanager (one receiver per topic, category routes), the
access token moves to its http_config as a credentials_file. Tags are
the one thing ntfy can't template, so the resolved marker is a title
prefix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

monitoring: scrape alertmanager, alert on failed notifications

validate / validate (pull_request) Successful in 25s

Details

7644a30700

With ntfy rendering at request time, a bad template surfaces only as a
400 on delivery. Nothing scraped Alertmanager before, so that failure
had no metric anywhere.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap force-pushed monitoring/drop-ntfy-bridge from 7644a30700

validate / validate (pull_request) Successful in 25s

Details

to cb067575b0

validate / validate (pull_request) Successful in 26s

Details

2026-10-08 00:15:37 -04:00

Compare

mysticalsoap force-pushed monitoring/drop-ntfy-bridge from cb067575b0

validate / validate (pull_request) Successful in 26s

Details

to 2de8ba0340