From 2b511590e6e874eabfe8c06e967a65a6deb101cc Mon Sep 17 00:00:00 2001 From: Sebastian Mendel Date: Wed, 15 Jul 2026 13:36:53 +0200 Subject: [PATCH 1/3] docs(ci): add runner-capacity diagnosis for slow pipelines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A slow GitHub Actions pipeline is often runner-acquisition-bound, not job-duration-bound, and the two need different fixes. New reference (references/ci-runner-capacity.md) documents: - the org-wide concurrency cap by plan (Free 20 / Team 60 / Enterprise 500), independent of repo visibility; - how to tell the regimes apart — Σ queue job-min vs Σ run job-min, and peak concurrency vs the cap — before optimizing; - why, when peak concurrency is below the cap, trimming job count buys ~0 wall clock (jobs wait for machines in parallel; fan-out is correct); - that the real fix there is capacity (plan upgrade / self-hosted, with the public-repo untrusted-PR-code caveat), while cutting compute still lowers org-wide queue pressure for every repo on the shared pool; - controlling for run-to-run queue noise by comparing runs at similar queue pressure. Indexed from SKILL.md. Came from /retro: yes Signed-off-by: Sebastian Mendel --- skills/github-project/SKILL.md | 1 + .../references/ci-runner-capacity.md | 79 +++++++++++++++++++ 2 files changed, 80 insertions(+) create mode 100644 skills/github-project/references/ci-runner-capacity.md diff --git a/skills/github-project/SKILL.md b/skills/github-project/SKILL.md index 92ab7d7..ad1ba30 100644 --- a/skills/github-project/SKILL.md +++ b/skills/github-project/SKILL.md @@ -105,6 +105,7 @@ State what a change does, not how good it is; no self-praise. See `references/no | Scorecard, CodeQL, security | `references/security-config.md` | | actionlint | `references/actionlint-guide.md` | | Workflow bash pitfalls | `references/workflow-bash-patterns.md` | +| Slow CI: runner capacity | `references/ci-runner-capacity.md` | | No editorializing | `references/no-editorializing.md` | | Fork merge base | `references/pr-commit-cleanup.md` | | Multi-repo batch ops | `references/multi-repo-operations.md` | diff --git a/skills/github-project/references/ci-runner-capacity.md b/skills/github-project/references/ci-runner-capacity.md new file mode 100644 index 0000000..4b8f15c --- /dev/null +++ b/skills/github-project/references/ci-runner-capacity.md @@ -0,0 +1,79 @@ +# CI Runner Capacity — Diagnose Before You Optimize + +A slow GitHub Actions pipeline is often **runner-acquisition-bound**, not +job-duration-bound. Before speeding up jobs or trimming the matrix, measure +which one you actually have — the fixes are different, and the common one +(trim job count) buys nothing under the common cause. + +## The concurrency cap + +GitHub-hosted runners are capped **org-wide**, by plan, regardless of repository +visibility (public repos get no relief): + +| Plan | Concurrent jobs (standard runners) | +|---|---| +| Free | 20 | +| Pro | 40 | +| Team | 60 | +| Enterprise | 500 | + +Source: . One PR push can +easily trigger 40–50 jobs across CI + checks + e2e, so a single PR on the Free +plan needs multiple full waves of the org's *entire* runner allotment — shared +with every other repo. + +## The two regimes, and why the distinction decides the fix + +- **Cap-bound** (peak concurrency ≈ the cap): jobs wait on *each other*. + Trimming job count and fanning out less both help. +- **Acquisition-bound** (peak concurrency **below** the cap, yet jobs still + queue): each job waits individually for GitHub to hand it a machine on the + shared pool. Job count is irrelevant — N jobs and N/2 jobs finish at the same + time because they wait in parallel. **Fan-out is correct here**; trimming the + matrix buys ≈0 wall clock. + +The trap: concluding "our jobs are slow, cut the matrix" when peak concurrency +was never near the cap. Measure first. + +## Measure it (one run) + +```bash +R=OWNER/REPO; RUN= +# Per-job queue (started-created) vs run (completed-started), in minutes: +gh api "repos/$R/actions/runs/$RUN/jobs?per_page=100" --jq ' + .jobs[] | select(.conclusion!=null) | + { q: (((.started_at|fromdateiso8601)-(.created_at|fromdateiso8601))/60), + r: (((.completed_at|fromdateiso8601)-(.started_at|fromdateiso8601))/60), + n: .name } | "queue=\(.q|floor)m run=\(.r|floor)m \(.n)"' +``` + +Then compute two totals and the peak: + +- **Σ queue job-min vs Σ run job-min.** If queue ≫ run (e.g. 415 vs 50), the + pipeline is waiting for machines, not computing. +- **Peak concurrent jobs** — sweep-line the `started_at`/`completed_at` events. + Below the plan cap ⇒ acquisition-bound. + +Control for noise: queue time swings run-to-run with whatever else the org is +running. Compare runs at *similar* queue pressure, not two arbitrary runs — a +single congested run makes any change look worthless. + +## What each regime's real fix is + +- **Acquisition-bound** — the lever is **capacity**, a business decision, not a + workflow edit: upgrade the plan (Free 20 → Team 60), or add self-hosted + runners. (Self-hosted on a **public** repo runs untrusted PR code — a known + hazard; gate on `pull_request` from forks.) Note: a higher *cap* does not + necessarily lower *acquisition latency* on the shared pool — verify before + spending. +- **Cap-bound** — trim job count (a matrix that removes a required-by-name + status check makes PRs un-mergeable, so change the ruleset first), gate + docs-only PRs, and add `concurrency: cancel-in-progress` for `pull_request` + (never for `merge_group`/push-to-main). + +## Compute is still worth cutting + +Even when it buys no wall clock, halving job-minutes lowers org-wide queue +pressure for *every* repo sharing the pool — so making jobs faster is the right +first move; just don't promise a wall-clock win it can't deliver under the +acquisition-bound regime. From 2dc03d878da74dcec1ea9cfd5cbe03778fae24c1 Mon Sep 17 00:00:00 2001 From: Sebastian Mendel Date: Wed, 15 Jul 2026 13:41:03 +0200 Subject: [PATCH 2/3] docs(ci): use a plain placeholder in the shell snippet Address review: in a bash block is a shell redirection if pasted. Use RUN_ID. Came from /retro: yes Signed-off-by: Sebastian Mendel --- skills/github-project/references/ci-runner-capacity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/github-project/references/ci-runner-capacity.md b/skills/github-project/references/ci-runner-capacity.md index 4b8f15c..563c3b4 100644 --- a/skills/github-project/references/ci-runner-capacity.md +++ b/skills/github-project/references/ci-runner-capacity.md @@ -38,7 +38,7 @@ was never near the cap. Measure first. ## Measure it (one run) ```bash -R=OWNER/REPO; RUN= +R=OWNER/REPO; RUN=RUN_ID # Per-job queue (started-created) vs run (completed-started), in minutes: gh api "repos/$R/actions/runs/$RUN/jobs?per_page=100" --jq ' .jobs[] | select(.conclusion!=null) | From 02552fdd4fb70cd7a3e0a1f068249066a6360ee0 Mon Sep 17 00:00:00 2001 From: Sebastian Mendel Date: Wed, 15 Jul 2026 13:45:07 +0200 Subject: [PATCH 3/3] docs: trim SKILL.md under the 500-word cap after adding the index row The runner-capacity index row pushed SKILL.md to 505 words (cap 500). Shorten the row label and drop the redundant one-line subtitle (it restated the frontmatter description). Signed-off-by: Sebastian Mendel --- skills/github-project/SKILL.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/skills/github-project/SKILL.md b/skills/github-project/SKILL.md index ad1ba30..f45280e 100644 --- a/skills/github-project/SKILL.md +++ b/skills/github-project/SKILL.md @@ -12,8 +12,6 @@ allowed-tools: Bash(gh:*) Bash(git:*) Bash(grep:*) Read Write # GitHub Project Skill -Repository configuration, troubleshooting, and collaboration. - ## When to Use - **Post `gh repo create` + initial push, before first PR** — apply branch protection (REQUIRED, see below) @@ -105,7 +103,7 @@ State what a change does, not how good it is; no self-praise. See `references/no | Scorecard, CodeQL, security | `references/security-config.md` | | actionlint | `references/actionlint-guide.md` | | Workflow bash pitfalls | `references/workflow-bash-patterns.md` | -| Slow CI: runner capacity | `references/ci-runner-capacity.md` | +| Runner capacity | `references/ci-runner-capacity.md` | | No editorializing | `references/no-editorializing.md` | | Fork merge base | `references/pr-commit-cleanup.md` | | Multi-repo batch ops | `references/multi-repo-operations.md` |