No egress shaping: a saturated uplink collapses downstream throughput #156

New issue

Open

opened 2026-08-21 12:11:48 -04:00 by mysticalsoap · 1 comment

mysticalsoap commented

2026-08-21 12:11:48 -04:00

Owner

The connection is heavily asymmetric and nothing shapes the upload, so any sustained outbound transfer starves downstream ACKs and collapses the whole link for every other consumer.

Measured 2026-08-21 from 30d of cadvisor counters on gluetun:

Direction	Observed
Down	~927 Mbps peak (115.89 MB/s)
Up	~43.5 Mbps hard ceiling (5.44 MB/s peak, p99 5.0–5.37)

The upload distribution sits flat on its own maximum rather than tailing off, which is a hard cap and not a workload artifact. Both directions cross the same tunnel, so the VPN isn't the constraint — the WAN link is simply that asymmetric.

Impact when the uplink is full: downstream drops to 84–175 KB/s, a ~99.98% degradation on a gigabit downlink. That's textbook bufferbloat — ACKs queue behind the upload. It took out CI repeatedly (mysticalsoap/aqomui#115) but affects everything: pulls, backups, streaming, any interactive use.

This is currently latent rather than fixed. The immediate trigger was public-tracker seeding, resolved by a qbit_manage share-limit change, but nothing prevents a restic run, a private-tracker seed spike, or any other sustained upload from doing it again.

Why the host doesn't already absorb it

enp5s0 has fq_codel on its hardware queues, but it drains at 1 Gbps into the LAN switch, so its queue never builds and fq_codel never engages. The buffer that actually fills is in the ISP modem, on the ~43 Mbps WAN egress.

Fix

Make the host the bottleneck instead of the modem, by shaping egress slightly below the real uplink rate so the queue forms somewhere with a competent AQM:

tc qdisc replace dev enp5s0 root cake bandwidth 38Mbit

Roughly 90–95% of measured line rate; needs tuning against a real bufferbloat test. This only governs traffic originating from this host — which is where essentially all the upload comes from, so it addresses the actual offender without touching the router.

To make it durable it needs a home: a systemd unit or a NetworkManager dispatcher script in dotfiles, applied on link-up. Router-side CAKE/SQM would additionally cover other devices on the LAN and is worth considering if the router supports it, but it isn't required to fix this.

Verification is blocked on a monitoring gap

There are no host WAN interface metrics at all. node-exporter runs in a container netns, so its device label only has eth0/lo — enp5s0 is absent and no node_network_* series exist for it. The figures above had to be inferred from a container's counters.

Without that, there's no way to confirm a shaper is working, no alert when the uplink pins, and no way to notice the next occurrence. Worth splitting into its own issue if it's more than a one-line fix.

The connection is heavily asymmetric and nothing shapes the upload, so any sustained outbound transfer starves downstream ACKs and collapses the whole link for every other consumer. Measured 2026-08-21 from 30d of cadvisor counters on `gluetun`: | Direction | Observed | |---|---| | Down | ~927 Mbps peak (115.89 MB/s) | | Up | **~43.5 Mbps hard ceiling** (5.44 MB/s peak, p99 5.0–5.37) | The upload distribution sits flat on its own maximum rather than tailing off, which is a hard cap and not a workload artifact. Both directions cross the same tunnel, so the VPN isn't the constraint — the WAN link is simply that asymmetric. **Impact when the uplink is full:** downstream drops to 84–175 KB/s, a ~99.98% degradation on a gigabit downlink. That's textbook bufferbloat — ACKs queue behind the upload. It took out CI repeatedly (`mysticalsoap/aqomui#115`) but affects everything: pulls, backups, streaming, any interactive use. This is currently latent rather than fixed. The immediate trigger was public-tracker seeding, resolved by a qbit_manage share-limit change, but nothing prevents a restic run, a private-tracker seed spike, or any other sustained upload from doing it again. ## Why the host doesn't already absorb it `enp5s0` has fq_codel on its hardware queues, but it drains at 1 Gbps into the LAN switch, so its queue never builds and fq_codel never engages. The buffer that actually fills is in the ISP modem, on the ~43 Mbps WAN egress. ## Fix Make the host the bottleneck instead of the modem, by shaping egress slightly below the real uplink rate so the queue forms somewhere with a competent AQM: ``` tc qdisc replace dev enp5s0 root cake bandwidth 38Mbit ``` Roughly 90–95% of measured line rate; needs tuning against a real bufferbloat test. This only governs traffic originating from this host — which is where essentially all the upload comes from, so it addresses the actual offender without touching the router. To make it durable it needs a home: a systemd unit or a NetworkManager dispatcher script in `dotfiles`, applied on link-up. Router-side CAKE/SQM would additionally cover other devices on the LAN and is worth considering if the router supports it, but it isn't required to fix this. ## Verification is blocked on a monitoring gap There are no host WAN interface metrics at all. node-exporter runs in a container netns, so its `device` label only has `eth0`/`lo` — `enp5s0` is absent and no `node_network_*` series exist for it. The figures above had to be inferred from a container's counters. Without that, there's no way to confirm a shaper is working, no alert when the uplink pins, and no way to notice the next occurrence. Worth splitting into its own issue if it's more than a one-line fix.

mysticalsoap added the

general-admin

label

2026-08-21 12:11:48 -04:00

mysticalsoap commented

2026-08-21 13:15:43 -04:00

Author

Owner

Deprioritise rather than cap

A flat bandwidth limit gives up capacity permanently. CAKE's diffserv4 gives the same protection without the waste — its Bulk tin has a 6.25% floor, not a ceiling, so torrents take the whole uplink when it's idle and collapse to ~6% the moment a Jellyfin stream wants it. No manual switching.

tc qdisc replace dev enp5s0 root cake bandwidth 38Mbit diffserv4 nat wash
iptables -t mangle -A FORWARD -s <gluetun> -j DSCP --set-dscp 0x01

0x01 is LE (RFC 8622) rather than CS1 — CAKE maps both to Bulk on this kernel, but CS1 is documented as inconsistent (prioritised over CS0 on some switches, deprioritised on wifi). wash clears the mark after classification so nothing nonstandard reaches the VPN provider.

This is the standard recipe, not an invention. From CAKE's own technical wiki: BitTorrent "should be directed to the Bulk class", and "the only way known to 'fix' bittorrent is to classify it somewhat, somehow, as 'background'." The same page recommends diffserv4, and notes "de-prioritization seems a good idea, prioritization not so much" — which is what this does. Nothing gets promoted, so nothing can be starved by mistake.

µTP does not make this redundant. qBittorrent enables it by default and LEDBAT targets 100ms added delay specifically so torrents yield (libtorrent, BEP 29). But it yields reliably only against TCP on the same bottleneck — peers connecting over plain TCP don't back off at all, and qBittorrent's own forum is blunt that it is not a magic bullet. The shaper is the backstop for what µTP structurally can't cover.

Correction to the original text

gluetun runs WireGuard, not OpenVPN (VPN_TYPE=wireguard, kernel implementation, peer 163.5.171.2:51820). That makes classification easier: all torrent traffic is a single outer UDP flow to one fixed endpoint, so CAKE's flow hashing already gives it one flow's share rather than letting hundreds of connections gang up. Meaningful improvement lands from the shaper alone, before any marking.

Blocked on mysticalsoap/aqomui#116

While aqomui is connected, container traffic is captured by its /1 route override and double-tunnelled — measured 111MB through gluetun reappearing as 118MB through tun_aqomui. Everything on enp5s0 is then one indistinguishable outer flow, and no DSCP marking inside it is visible to a shaper.

The shaper still helps whenever the host VPN is down, but classification only works once #116 is resolved.

## Deprioritise rather than cap A flat `bandwidth` limit gives up capacity permanently. CAKE's `diffserv4` gives the same protection without the waste — its **Bulk** tin has a 6.25% *floor*, not a ceiling, so torrents take the whole uplink when it's idle and collapse to ~6% the moment a Jellyfin stream wants it. No manual switching. ``` tc qdisc replace dev enp5s0 root cake bandwidth 38Mbit diffserv4 nat wash iptables -t mangle -A FORWARD -s <gluetun> -j DSCP --set-dscp 0x01 ``` `0x01` is LE (RFC 8622) rather than CS1 — CAKE maps both to Bulk on this kernel, but CS1 is documented as inconsistent (prioritised over CS0 on some switches, deprioritised on wifi). `wash` clears the mark after classification so nothing nonstandard reaches the VPN provider. This is the standard recipe, not an invention. From CAKE's own [technical wiki](https://www.bufferbloat.net/projects/codel/wiki/CakeTechnical/): BitTorrent "should be directed to the Bulk class", and *"the only way known to 'fix' bittorrent is to classify it somewhat, somehow, as 'background'."* The same page recommends `diffserv4`, and notes **"de-prioritization seems a good idea, prioritization not so much"** — which is what this does. Nothing gets promoted, so nothing can be starved by mistake. **µTP does not make this redundant.** qBittorrent enables it by default and LEDBAT targets 100ms added delay specifically so torrents yield ([libtorrent](https://www.libtorrent.org/utp.html), [BEP 29](https://www.bittorrent.org/beps/bep_0029.html)). But it yields reliably only against TCP on the same bottleneck — peers connecting over plain TCP don't back off at all, and qBittorrent's own forum is blunt that it is [not a magic bullet](https://forum.qbittorrent.org/viewtopic.php?t=3835). The shaper is the backstop for what µTP structurally can't cover. ## Correction to the original text gluetun runs **WireGuard**, not OpenVPN (`VPN_TYPE=wireguard`, kernel implementation, peer `163.5.171.2:51820`). That makes classification easier: all torrent traffic is a *single* outer UDP flow to one fixed endpoint, so CAKE's flow hashing already gives it one flow's share rather than letting hundreds of connections gang up. Meaningful improvement lands from the shaper alone, before any marking. ## Blocked on mysticalsoap/aqomui#116 While aqomui is connected, container traffic is captured by its `/1` route override and double-tunnelled — measured 111MB through gluetun reappearing as 118MB through `tun_aqomui`. Everything on `enp5s0` is then one indistinguishable outer flow, and no DSCP marking inside it is visible to a shaper. The shaper still helps whenever the host VPN is down, but classification only works once #116 is resolved.