CI downloads 174 MB before it runs a single test #115

New issue

Closed

opened 2026-08-21 01:40:51 -04:00 by mysticalsoap · 1 comment

mysticalsoap commented

2026-08-21 01:40:51 -04:00

Owner

The test job installs its dependencies with a full pacman -Syu on archlinux:latest at the start of every run — currently 174 MB of downloads, all of it before actions/checkout has even run. The runner shares this host's internet connection, so whenever that connection is busy the job dies during package retrieval with error: failed retrieving file ... Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds, having executed none of the code it was meant to test.

Three of the last five runs on PR #114 failed this way (136 and 138 failed, 137 passed with identical dependency needs) — the outcome tracks link conditions, not the diff. Measured during the failures: ~84–175 KB/s from three different Arch mirrors, with the host's uplink saturated at ~5 MB/s by seeding, which starves downstream ACKs. Related but distinct from the DNS-starvation flake already recorded for this runner.

CI is disabled for now (on: workflow_dispatch only) so PRs stop collecting red Xs for reasons unrelated to their contents. Re-enable as part of fixing this.

Options, roughly best-first:

Bake the dependencies into a runner image. Build once from the PKGBUILD dependency list, push to a local registry, and have the job runs-on that image. Per-run download drops to ~zero, and the job stops being hostage to the connection. Costs a rebuild step when dependencies change, which is rare and explicit.
Persist the pacman cache between runs. Cheaper to implement, but the first run after any upstream package bump pays the full cost again, so it narrows the window without closing it.
Pin the base image tag rather than :latest, so -Syu has less to reconcile. Helps a little, and pins the tested platform, which is arguably worth having on its own.

Worth deciding alongside: the job tests against a rolling :latest Arch, so an upstream breakage and a real regression currently look identical.

The `test` job installs its dependencies with a full `pacman -Syu` on `archlinux:latest` at the start of every run — currently **174 MB** of downloads, all of it before `actions/checkout` has even run. The runner shares this host's internet connection, so whenever that connection is busy the job dies during package retrieval with `error: failed retrieving file ... Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds`, having executed none of the code it was meant to test. Three of the last five runs on PR #114 failed this way (136 and 138 failed, 137 passed with identical dependency needs) — the outcome tracks link conditions, not the diff. Measured during the failures: ~84–175 KB/s from three different Arch mirrors, with the host's uplink saturated at ~5 MB/s by seeding, which starves downstream ACKs. Related but distinct from the DNS-starvation flake already recorded for this runner. **CI is disabled for now** (`on: workflow_dispatch` only) so PRs stop collecting red Xs for reasons unrelated to their contents. Re-enable as part of fixing this. Options, roughly best-first: - **Bake the dependencies into a runner image.** Build once from the PKGBUILD dependency list, push to a local registry, and have the job `runs-on` that image. Per-run download drops to ~zero, and the job stops being hostage to the connection. Costs a rebuild step when dependencies change, which is rare and explicit. - **Persist the pacman cache between runs.** Cheaper to implement, but the first run after any upstream package bump pays the full cost again, so it narrows the window without closing it. - **Pin the base image tag** rather than `:latest`, so `-Syu` has less to reconcile. Helps a little, and pins the tested platform, which is arguably worth having on its own. Worth deciding alongside: the job tests against a rolling `:latest` Arch, so an upstream breakage and a real regression currently look identical.

mysticalsoap referenced this issue from a commit

2026-08-21 01:41:37 -04:00

chore: run CI manually only, until it stops being link-bound (#115)

mysticalsoap referenced this issue from a commit

2026-08-21 11:44:26 -04:00

fix: pin the CI platform and cache packages, re-enable triggers (#115)

mysticalsoap commented

2026-08-21 11:58:26 -04:00

Author

Owner

Correction: the download size wasn't the cause

The diagnosis above blames the ~174MB of package downloads. That framing is wrong in a way worth recording, because it made "shrink the download" look like the fix when the actual problem was elsewhere.

174MB is about two seconds on this link. The runs failed because the downlink collapsed to 84–175 KB/s — a ~99.98% degradation — and that is ordinary bufferbloat from a saturated uplink starving ACKs. Anything sharing the connection was affected; CI was just the thing that reported it as a red X.

Measured from 30d of container counters:

Direction	Observed
Down	~927 Mbps peak (115.89 MB/s)
Up	~43.5 Mbps hard ceiling (5.44 MB/s peak, p99 5.0–5.37)

The upload distribution sits flat on its own maximum, which is a hard cap rather than a workload tail. Both directions cross the same tunnel, so the VPN isn't the constraint — the WAN link is simply that asymmetric. Nothing on the host can absorb it either: enp5s0 has fq_codel, but it drains at 1 Gbps into the switch so its queue never builds. The buffer that fills is in the ISP modem, downstream of anything we control.

What was actually saturating it: a share_limits group in qbit_manage that had matched nothing since it was written. It required both the noHL and Public tags, but an imported release keeps its library hardlink for life, so noHL never arrives and the group sat at Torrents (0) run after run without erroring. Public torrents therefore had no share limit at all — 33 of them, seeding up to 83 days each, pinning the uplink at a sustained ~4.5 MB/s. Fixed by dropping the hardlink gate; upload fell to 0.18 MB/s.

What shipped here

The CI change stands on its own merits regardless of link state, so it was done anyway — but as an efficiency and reproducibility fix, not as the remedy for the failures.

Of the three options above, the second turned out to be the strongest once combined with the third. Package caching was dismissed as only "narrowing the window", but that assumed :latest stayed. Pinning the base image and pointing pacman at a frozen archive.archlinux.org snapshot closes it: versions can't move underneath the cache, so a run downloads nothing but the repo databases.

178MB → 8.7MB per run, zero package downloads on reruns
Both the image tag and the snapshot pin move only when moved deliberately, together — which resolves the last point above, that upstream breakage and a real regression looked identical
Dependencies now come from the full PKGBUILD depends array rather than the python- subset, so CI exercises what an install actually gets
push/pull_request triggers restored

Baking a prebuilt image was considered and rejected: every path to building one needed privileges worth more than the problem. The Dagu socket proxy withholds POST /build deliberately, and building in CI would have meant granting jobs Docker access permanently — which Forgejo's own docs note has no security isolation — to avoid a weekly task that needs it once.

Verified in the pinned container before merge: 274 passed, ruff and compileall clean. Run 139 on trunk added nothing to the package cache, confirming it was served entirely from it.

## Correction: the download size wasn't the cause The diagnosis above blames the ~174MB of package downloads. That framing is wrong in a way worth recording, because it made "shrink the download" look like the fix when the actual problem was elsewhere. **174MB is about two seconds on this link.** The runs failed because the *downlink collapsed to 84–175 KB/s* — a ~99.98% degradation — and that is ordinary bufferbloat from a saturated uplink starving ACKs. Anything sharing the connection was affected; CI was just the thing that reported it as a red X. Measured from 30d of container counters: | Direction | Observed | |---|---| | Down | ~927 Mbps peak (115.89 MB/s) | | Up | **~43.5 Mbps hard ceiling** (5.44 MB/s peak, p99 5.0–5.37) | The upload distribution sits flat on its own maximum, which is a hard cap rather than a workload tail. Both directions cross the same tunnel, so the VPN isn't the constraint — the WAN link is simply that asymmetric. Nothing on the host can absorb it either: `enp5s0` has fq_codel, but it drains at 1 Gbps into the switch so its queue never builds. The buffer that fills is in the ISP modem, downstream of anything we control. **What was actually saturating it:** a `share_limits` group in qbit_manage that had matched nothing since it was written. It required both the `noHL` and `Public` tags, but an imported release keeps its library hardlink for life, so `noHL` never arrives and the group sat at `Torrents (0)` run after run without erroring. Public torrents therefore had no share limit at all — 33 of them, seeding up to 83 days each, pinning the uplink at a sustained ~4.5 MB/s. Fixed by dropping the hardlink gate; upload fell to 0.18 MB/s. ## What shipped here The CI change stands on its own merits regardless of link state, so it was done anyway — but as an efficiency and reproducibility fix, not as the remedy for the failures. Of the three options above, the second turned out to be the strongest once combined with the third. Package caching was dismissed as only "narrowing the window", but that assumed `:latest` stayed. Pinning the base image *and* pointing pacman at a frozen `archive.archlinux.org` snapshot closes it: versions can't move underneath the cache, so a run downloads nothing but the repo databases. - **178MB → 8.7MB per run**, zero package downloads on reruns - Both the image tag and the snapshot pin move only when moved deliberately, together — which resolves the last point above, that upstream breakage and a real regression looked identical - Dependencies now come from the full PKGBUILD `depends` array rather than the `python-` subset, so CI exercises what an install actually gets - `push`/`pull_request` triggers restored Baking a prebuilt image was considered and rejected: every path to building one needed privileges worth more than the problem. The Dagu socket proxy withholds `POST /build` deliberately, and building in CI would have meant granting jobs Docker access permanently — which Forgejo's own docs note has no security isolation — to avoid a weekly task that needs it once. Verified in the pinned container before merge: 274 passed, ruff and compileall clean. Run 139 on trunk added nothing to the package cache, confirming it was served entirely from it.

mysticalsoap closed this issue

2026-08-21 11:58:37 -04:00