vps: crowdsec firewall bouncer, drop the traefik bait and catchall routers

mysticalsoap commented

2026-09-21 01:22:24 -04:00

Owner

Problem

#102. Bans decided at home were enforced only by the Traefik plugin, so the git-over-SSH port (27341) had no per-IP enforcement at all, and a banned scanner still completed TCP and TLS handshakes to home before getting a 403. The bait router, the catchall router and addInternals existed only to make that L7 plugin see unrouted traffic.

Fix

VPS: crowdsec-firewall-bouncer-nftables from CrowdSec's apt repo (any suite, signing key inline in a deb822 sources file), installed by install.sh. Config is a committed .yaml.local overlay plus an api key that lives only on the box; without the key file the bouncer stays down. Bouncer only, no agent (commit body has the reasoning).
Home: a lapi listener on the tunnel HAProxy (8081 to crowdsec:8080, tunnel peer only), the wireguard container joins the crowdsec network, and a second BOUNCER_KEY_* secret registers the VPS bouncer at container start.
Traefik: bait and catchall routers and addInternals removed. Traefik logs unrouted requests regardless (the rotated access logs from before the routers existed carry thousands of unrouted 404s) and the bait scenario keys on the path, so detection is unchanged; enforcing those bans is now the VPS's job. The self-response whitelist parser stays (geoblock 451, AppSec 403, plugin 403 inside the pull window).
Docs: Public-Exposure, Auth-and-Security, setup/VPS, setup/Infrastructure, Maintenance.

Verification

Static: _helper/ci/check-compose.sh passes for every stack, haproxy -c on the edited home config passes inside the running haproxy container, bash -n on install.sh.

Live:

just up infra on the branch: the entrypoint logged Registered bouncer for vps; wireguard is on the crowdsec network.
just vps-deploy: the package installed from the any suite and the bouncer came up with the key. At home cscli bouncers list shows vps (type crowdsec-firewall-bouncer) from 172.18.4.3, the tunnel end's crowdsec-network address, pulling /v1/decisions/stream every 10s with 200s in the LAPI log.
The stream the VPS receives, queried with its own key from inside the container (new array only): 23,742 CAPI decisions and 34 local ones, deduplicated by the LAPI to the longest decision per IP, which is why most local bans travel as CAPI. go-cs-bouncer never sets community_pull, so the request carries community_pull=false; this LAPI serves the same set with or without it, so the community blocklist is enforced at the edge too.
Sets are per origin in v0.0.36 (crowdsec-blacklists-CAPI, -crowdsec, -cscli); the base crowdsec-blacklists set stays empty. The bouncer's usage metrics at home show 23.74k active CAPI IPs and packets already dropped. Its active_decisions gauge is reported for one origin only (collectActiveBannedIPs returns inside its loop, v0.0.36), so the missing rows for the other origins mean nothing; their per-origin drop counters exist, so their rules do.
Delta-stream drop bug upstream (cs-firewall-bouncer #508) was fixed on the LAPI side by crowdsec PR 4620 in 1.8.0; this LAPI is 1.8.1.
VPS side, read with sudo by the user: service active since 05:47:22, log shows 23784 decisions added on the startup sync with sets and rules created for origins CAPI (v4 and v6) and crowdsec, then cscli on the first manual test ban. Set sizes at 06:2x: crowdsec-blacklists-CAPI 23,229, -crowdsec 34, -cscli 1, crowdsec6-blacklists-CAPI 513. Test ban 203.0.113.8 (added at home 06:22:13) was in the cscli set by 06:22:22; the earlier 30m one appeared at 05:49:42 and left at 06:19:42 on expiry.

Not exercised: a bait-path probe from outside against the raw VPS IP. Detection is unchanged by this PR: unrouted 404s were in the access log before the routers existed.

## Problem #102. Bans decided at home were enforced only by the Traefik plugin, so the git-over-SSH port (27341) had no per-IP enforcement at all, and a banned scanner still completed TCP and TLS handshakes to home before getting a 403. The bait router, the catchall router and `addInternals` existed only to make that L7 plugin see unrouted traffic. ## Fix - VPS: `crowdsec-firewall-bouncer-nftables` from CrowdSec's apt repo (`any` suite, signing key inline in a deb822 sources file), installed by `install.sh`. Config is a committed `.yaml.local` overlay plus an api key that lives only on the box; without the key file the bouncer stays down. Bouncer only, no agent (commit body has the reasoning). - Home: a `lapi` listener on the tunnel HAProxy (8081 to `crowdsec:8080`, tunnel peer only), the `wireguard` container joins the `crowdsec` network, and a second `BOUNCER_KEY_*` secret registers the VPS bouncer at container start. - Traefik: bait and catchall routers and `addInternals` removed. Traefik logs unrouted requests regardless (the rotated access logs from before the routers existed carry thousands of unrouted 404s) and the bait scenario keys on the path, so detection is unchanged; enforcing those bans is now the VPS's job. The self-response whitelist parser stays (geoblock 451, AppSec 403, plugin 403 inside the pull window). - Docs: Public-Exposure, Auth-and-Security, setup/VPS, setup/Infrastructure, Maintenance. ## Verification Static: `_helper/ci/check-compose.sh` passes for every stack, `haproxy -c` on the edited home config passes inside the running `haproxy` container, `bash -n` on `install.sh`. Live: - `just up infra` on the branch: the entrypoint logged `Registered bouncer for vps`; `wireguard` is on the `crowdsec` network. - `just vps-deploy`: the package installed from the `any` suite and the bouncer came up with the key. At home `cscli bouncers list` shows `vps` (type `crowdsec-firewall-bouncer`) from 172.18.4.3, the tunnel end's crowdsec-network address, pulling `/v1/decisions/stream` every 10s with 200s in the LAPI log. - The stream the VPS receives, queried with its own key from inside the container (`new` array only): 23,742 CAPI decisions and 34 local ones, deduplicated by the LAPI to the longest decision per IP, which is why most local bans travel as CAPI. go-cs-bouncer never sets `community_pull`, so the request carries `community_pull=false`; this LAPI serves the same set with or without it, so the community blocklist is enforced at the edge too. - Sets are per origin in v0.0.36 (`crowdsec-blacklists-CAPI`, `-crowdsec`, `-cscli`); the base `crowdsec-blacklists` set stays empty. The bouncer's usage metrics at home show 23.74k active CAPI IPs and packets already dropped. Its `active_decisions` gauge is reported for one origin only (`collectActiveBannedIPs` returns inside its loop, v0.0.36), so the missing rows for the other origins mean nothing; their per-origin drop counters exist, so their rules do. - Delta-stream drop bug upstream (cs-firewall-bouncer #508) was fixed on the LAPI side by crowdsec PR 4620 in 1.8.0; this LAPI is 1.8.1. - VPS side, read with sudo by the user: service active since 05:47:22, log shows `23784 decisions added` on the startup sync with sets and rules created for origins `CAPI` (v4 and v6) and `crowdsec`, then `cscli` on the first manual test ban. Set sizes at 06:2x: `crowdsec-blacklists-CAPI` 23,229, `-crowdsec` 34, `-cscli` 1, `crowdsec6-blacklists-CAPI` 513. Test ban `203.0.113.8` (added at home 06:22:13) was in the `cscli` set by 06:22:22; the earlier 30m one appeared at 05:49:42 and left at 06:19:42 on expiry. Not exercised: a bait-path probe from outside against the raw VPS IP. Detection is unchanged by this PR: unrouted 404s were in the access log before the routers existed.

mysticalsoap added 2 commits

2026-09-21 01:22:24 -04:00

vps: crowdsec firewall bouncer enforcing the home LAPI's decisions eae2feaf64

The git-over-SSH port had no per-IP enforcement at all: the tunnel hides
the client from everything at home, so a homelab ban never touched it. The
VPS is the one hop that sees real addresses natively, so enforcement moves
there: a firewall bouncer pulls decisions from the home LAPI through a new
listener on the home HAProxy, open to the tunnel peer only, and drops
matches in nftables ahead of ufw. A banned address now stops before the
TCP handshake instead of after a TLS handshake and a 403 from the Traefik
plugin, on every public port.

Bouncer only, no agent. The VPS sshd is publickey-only and nothing on the
box sees Forgejo's auth outcomes, so an agent there would add a write path
to the LAPI (a compromised VPS could ban arbitrary addresses) and detect
nothing worth having.

A native package rather than a Quadlet: writing host nftables needs the
host namespace and NET_ADMIN either way, so a container would keep the
packaging and lose the isolation. Debian dropped its own package after
bookworm and CrowdSec's repo has no trixie suite; the sources file points
at the repo's distro-agnostic `any` suite with the signing key inline,
which keeps curl and gpg off the box. The key is the one value that lives
only on the VPS, like the WireGuard private key; the rest of the config is
a .local overlay merged over the package default, so an upgrade never hits
a conffile prompt.

The home side registers the key from a docker secret at container start,
the mechanism the Traefik plugin's key already uses, which is why any
random value works and why rotation is delete-then-recreate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

traefik: drop the bait and catchall routers

validate / validate (pull_request) Successful in 27s

Details

69904a6568

Both existed to push traffic no router matched through the entrypoint
middleware chain so the CrowdSec plugin could act on it, and addInternals
existed so their @internal-routed requests reached the access log. The
VPS firewall bouncer enforces bans below Traefik, so an unrouted request
no longer needs a router to be blocked, and detection never depended on
one: Traefik logs unrouted requests as 404s regardless (the rotated
access logs from before the routers existed carry thousands of them), and
the bait scenario keys on the path alone. Dropping addInternals also keeps
the ping and prometheus scrape lines out of the access log.

The self-response whitelist parser stays: geoblock still answers 451 in
Traefik, AppSec still answers 403, and so does the plugin during the ten
seconds before the VPS bouncer's next pull.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap referenced this pull request

2026-09-21 01:22:24 -04:00

VPS: CrowdSec agent + firewall bouncer #102

mysticalsoap added 1 commit

2026-09-21 01:44:14 -04:00

docs: vps bouncer name and key copy command

validate / validate (pull_request) Successful in 25s

Details

6099e8c299

The crowdsec entrypoint names a bouncer after what follows BOUNCER_KEY_,
so the new one is `vps`. And sudo on the VPS prompts for a password, which
needs the terminal, so the key can't ride in on the same ssh's stdin: copy
it to the admin home first, then install it with a tty.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mysticalsoap force-pushed vps-firewall-bouncer from 6099e8c299

validate / validate (pull_request) Successful in 25s

Details

to 76b95db20a

validate / validate (pull_request) Successful in 25s

Details

2026-09-21 01:48:35 -04:00

Compare

mysticalsoap changed title from ~~WIP: vps: crowdsec firewall bouncer, drop the traefik bait and catchall routers~~ to vps: crowdsec firewall bouncer, drop the traefik bait and catchall routers

2026-09-21 01:51:07 -04:00

mysticalsoap force-pushed vps-firewall-bouncer from 76b95db20a

validate / validate (pull_request) Successful in 25s

Details

to 114c3c0679

validate / validate (pull_request) Successful in 23s

Details

2026-09-21 02:23:01 -04:00

Compare

mysticalsoap merged commit 4000f14bef into trunk

2026-10-01 12:50:49 -04:00

mysticalsoap deleted branch vps-firewall-bouncer