Skip to content

Audit that every merge on main went through the merge queue - #694

Merged
Birfy merged 1 commit into
mainfrom
feat/689-audit-merges
Oct 22, 2026
Merged

Birfy merged 1 commit into
mainfrom
feat/689-audit-merges

Conversation

@Birfy

@Birfy Birfy commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Written-by: Dev

Adds .github/scripts/audit-merges.mjs: a read-only check that every merge on main went through the merge queue, and exits 1 naming the ones that did not.

Fixes #689

Tip SHA: 2372531

Reproduce

#650 and #680 are build commits whose Merge gate run had failed, and main was pushed to them anyway — the runs API, live:

#680  run 39730969259  Merge gate  failure  2026-10-22T05:57:11Z  head 6b9820bb  branch gh-readonly-queue/main/pr-680-8e80237b
      main moved to 6b9820bb at 05:58:58Z, merged_by Birfy
#650  run 39653640861  Merge gate  failure  2026-10-21T18:32:27Z  head 6438cb2b  branch gh-readonly-queue/main/pr-650-fe79de66
      main moved to 6438cb2b at 18:34:05Z, merged_by Birfy
#672  run 39720497176  Merge gate  success  2026-10-22T04:06:23Z  head 93e52347
#674  run 39722858008  Merge gate  success  2026-10-22T04:31:32Z  head 9393d558

Nothing in the repository tells the two shapes apart: the only merge_group record of the landed commit is refused, and no check looks at what landed.

One correction to the issue's step 4, because a fixture pins it: head_sha= on the runs endpoint returns every workflow's run for that entry. On #680 the Test run of the same entry is success (39730969253), so "at least one run with conclusion === 'success'" would call that bypass a queue merge. The check therefore matches the Merge gate workflow (path === .github/workflows/merge-gate.yml), and a green Test run proves nothing about the gate.

What changed

.github/scripts/audit-merges.mjs (new, ~200 lines, no new dependency), its cases and four fixtures:

  • auditCommit({sha, pulls, runs}) — one commit in, one finding or null out. No pull request → direct commit: <sha>. A successful Merge gate run whose head_sha is this commit → queue merge. Otherwise bypassed queue: #<pr> <sha12> (merge_group: failure|absent), where absent means the gate never judged this commit and failure means it did and did not pass.
  • audit({commits, pullsFor, runsFor}) — the walk, over data already fetched; main({api, repo, limit}) — the CLI entry point the same shape as merge-gate.mjs (GH_REPO, GH_TOKEN, --limit, default 50), printing the lines and exiting 1 when there is a finding.
  • Read-only: no comment, label, merge or ref push. It walks /commits?sha=main, /commits/{sha}/pulls, /actions/runs?event=merge_group&head_sha={sha}.

The line shape is fixed because two readers depend on it (the Maintainer patrol and a hand run). A direct push is reported on its own line and is not counted as a bypass.

What I ran

At this tip, on the extracted tree:

  • TZ=UTC node --test .github/scripts/audit-merges.test.mjs → 9/9
  • TZ=UTC node --test ".github/scripts/*.test.mjs" → 173/173

Against the real repository, through this script's own main() with its api injected over the same REST paths gh api uses (gh is not installed on this box), at main b5de55d0, --limit 40:

bypassed queue: #680 6b9820bbd753 (merge_group: failure)
bypassed queue: #636 8e80237bafae (merge_group: failure)
bypassed queue: #650 6438cb2b63ec (merge_group: failure)
...
audit-merges: 40 commits on main, 31 bypassed the queue, 0 direct
exit code: 1

#672 and #674 produce no line, which is the acceptance condition. The older entries read absent because they predate the queue (with --limit 50: 41 bypassed, 0 direct, exit 1).

Red without the source change

Three mutations of the guard, each reverting only audit-merges.mjs:

  • the success test becomes "any merge_group run exists on the commit" (the issue's literal step 4) → 5 of 9 red, including the #680 fixture: the green Test run is read as the gate passing.
  • the workflow filter is dropped, keeping conclusion === 'success' → 4 of 9 red, same bypass fixture.
  • the head_sha match is dropped → 1 red, the "a green gate run on a different commit says nothing about this one" guard.

Restored → 9/9.

Not run

  • CI at this tip has not finished yet; no workflow runs this script (wiring it into a routine is out of scope in the issue).
  • gh CLI is absent on this box, so the live run above used the injected api rather than the real gh api; the paths are identical and the CI job that would call it has gh on the runner.
  • The suite's four ds-bot-patches cases pass here (173/173); in a tree without git they are the known four reds.

Risk

T3 — .github/**, per .github/agent/risk-rules.json; needs a PASS at this tip and a maintainer approve.

Docs-not-needed: a read-only .github/scripts check and its cases; no subsystem doc owns .github/.

#650 (6438cb2) and #680 (6b9820b) landed as build commits whose Merge gate run had failed,
so "only the queue merges" has been a convention that nothing checks. audit-merges.mjs walks the
last commits on main and prints `bypassed queue: #<pr> <sha> (merge_group: failure|absent)` when
the landed commit carries no successful gate run of its own, `direct commit: <sha>` for a push
that came from no pull request, and exits 1 when it found one. Read-only.

The gate run is matched by workflow, not by "some merge_group run succeeded": on #680 the Test
run of the very entry the gate refused is green, so the looser test would call that bypass a
queue merge.

Fixes #689

Docs-not-needed: adds a read-only .github/scripts check and its cases; no subsystem doc owns .github/.
@Birfy Birfy added the agent-pr Pull request opened by an agent label Oct 22, 2026
@Birfy Birfy self-assigned this Oct 22, 2026
@Birfy

Birfy commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at 2372531
Written-by: Reviewer

This does what #689 asked, and it does it on evidence rather than on a guess: the gate run is identified by workflow, not by "some merge_group run on this commit succeeded", and I confirmed that distinction matters — on #680 the Test run of the same queue entry is green while its Merge gate run failed, so matching any green run would have called the bypass a queue merge. It is read-only (GETs only, no comment, label, merge or ref push), and the four fixtures carry the real run ids of #680 and #674, not invented ones.

Ran:

Not run:

  • gh end to end: no gh on this machine, so the walk was driven through audit() with live API data instead of main()'s gh api calls; main() itself is exercised by the fixtures
  • nothing else: no runtime code is touched, and the tip's tests-ok and package jobs are green

Non-blocking:

  1. Nothing runs it yet. There is no step in .github/workflows/ and no line in AGENTS.md or docs/ that mentions audit-merges, so the patrol this was written for has nothing to discover. One step in nightly.yml, or one line in the routine's doc, is what turns it from a script into a check.
  2. On this repository the first run exits 1 with 14 of the last 25 commits. That is the truth about the history, but it also means the exit code can never be a clean signal until those commits age out of --limit 50; a --since/watermark (or a recorded baseline of known bypasses) would let a red run mean "a new bypass", which is what a patrol can act on.
  3. It proves "main moved to a commit whose gate run failed or is missing", not "the queue performed this merge": a green build commit pushed to main by hand passes. That is exactly the incident shape that was observed, so it is the right check to ship, but the header comment's "a commit with a successful Merge gate run on it went through the queue" is one claim stronger than what the data can show.
  4. main()'s comment says api, env and out are injectable, but there is no out parameter, and the injection has to be synchronous — fetchJson is never awaited, so an async client answers "the commits API did not return a list". I hit that myself; one clarifying line in the comment would save the next caller the same ten minutes.
  5. A merge_group run that is still going has no conclusion, so gateRan is true and the line says failure while the run is in progress. running as a third state, or skipping the commit until its run concludes, reads more honestly.

T3: awaiting a maintainer approve at this tip; I am not enqueueing it.

@Birfy Birfy added the awaiting-approval PR is green and waits for the maintainer's approval (T2/T3) or a review PASS at its tip label Oct 22, 2026
@Birfy
Birfy added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit aada059 Oct 22, 2026
9 checks passed
@Birfy
Birfy deleted the feat/689-audit-merges branch October 22, 2026 07:26
@Birfy Birfy removed the awaiting-approval PR is green and waits for the maintainer's approval (T2/T3) or a review PASS at its tip label Oct 22, 2026
@Birfy Birfy removed the agent-pr Pull request opened by an agent label Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Audit that every merge on main went through the merge queue

1 participant