Skip to content

audit-merges: a watermark, and a read-only patrol that runs it - #708

Closed
Birfy wants to merge 1 commit into
mainfrom
fix/697-audit-watermark
Closed

Birfy wants to merge 1 commit into
mainfrom
fix/697-audit-watermark

Conversation

@Birfy

@Birfy Birfy commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Written-by: Dev

Reproduce

At cbce2b03 the audit is correct and unusable: over the last 40 commits on main it exits 1 on 26 findings, every one of them history from before merge-gate became a required check, and nothing runs it — no workflow in .github/workflows/ references it, so the patrol it was written for does not exist. A check that is red on its first run and stays red is a check nobody reads.

$ node .github/scripts/audit-merges.mjs --limit 40        # live API; this box has no gh, so curl-less fetch harness
bypassed queue: #680 6b9820bbd753 (merge_group: failure)
bypassed queue: #636 8e80237bafae (merge_group: failure)
…
audit-merges: 40 commits on main, 26 bypassed the queue, 0 direct, 0 earlier than the watermark
$ echo $?
1

The same run with a watermark at the current head (cbce2b03, 2026-10-22T07:29:05Z) walks 2 commits and exits 0, and a baseline run over 50 commits reports 36 findings without failing — the two shapes this PR puts into a patrol.

What changed

  • --since <sha|ISO date>. A sha is resolved through GET /commits/{sha} to its committer date; the walk is then fetched with since=. A finding at or before the watermark prints as earlier bypass: #<pr> <commit> (merge_group: failure|absent) or earlier direct commit: <commit> and does not affect the exit code, so exit 1 now means a bypass newer than the watermark. The boundary commit itself counts as earlier. An unreadable --since is an error, not a silent unfiltered walk, and no --since keeps the old behaviour for a hand run.
  • --baseline. The first run of a patrol that has no previous run: it walks the window unfiltered and marks every finding earlier, so the history it finds becomes the watermark instead of a red first run. It is exclusive with --since — giving both is an error, not one silently winning.
  • .github/workflows/audit-merges.yml — the read-only patrol: schedule plus workflow_dispatch, permissions: contents: read, actions: read, pull-requests: read, the script taken from main, and the watermark is the head_sha of its own previous successful run (GET /actions/workflows/audit-merges.yml/runs?status=success&per_page=1), so no file has to be committed and the job cannot write. With no previous run it passes --baseline.
  • docs/autonomous-dev.md (Merge plumbing) gains one paragraph naming the patrol and what a red run means.
  • Header comment corrections — two of the notes left on Audit that every merge on main went through the merge queue #694: it said a commit whose gate run failed "was nevertheless pushed to main", which is more than the data shows (the pull request names it as its merge commit), and it offered an out parameter that does not exist.

What I ran

At the tip (the local tree is the pushed blob set):

$ TZ=UTC node --test ".github/scripts/*.test.mjs"
# tests 180
# pass 180
$ TZ=UTC node --test .github/scripts/audit-merges.test.mjs
# tests 16
# pass 16        # 7 of them new: watermark (2), sha resolution, unreadable watermark, baseline, exclusivity, patrol shape

Live runs against the real repository (GitHub API only; no gh, no curl on this box — the harness pre-fetches the same paths the script asks for):

run commits findings exit
no watermark, --limit 40 40 26 bypassed queue 1 (unchanged behaviour)
--since cbce2b03, --limit 40 2 0 0
--since 2026-10-22T00:00:00Z, --limit 40 28 14 bypassed queue 1 — red means new
--baseline, --limit 50 50 36 earlier bypass 0 — first green is the baseline

Red without the source change

  • Base audit-merges.mjs with the new test file: 6 of 16 red (both watermark cases, sha resolution, unreadable watermark, baseline, exclusivity). The patrol-shape case reads the new workflow, so it is part of the same change.
  • Mutations on this tip: drop the earlier marking → 2 red; make exitCodeFor ignore the watermark → 1 red; drop the baseline marking → 1 red; drop the exclusivity check → 1 red; remove --since/--baseline from the patrol's run step → 1 red.

Not run

  • The workflow has never executed on GitHub — a branch cannot workflow_dispatch. Its shape is held by a text test (read-only permissions, both triggers, the watermark source, both invocation paths), and a first real run will exercise --baseline by construction. Nothing was dispatched.
  • No npm run verify, smoke test or bench: the change touches .github/scripts, a workflow and one docs paragraph, none of which the core suite loads.
  • The queue was not touched, nothing was enqueued and nothing merged.

Risk

T3 — .github/**. The change is read-only by construction: the script only reads the API, and the workflow holds contents: read, actions: read, pull-requests: read and nothing else, so it cannot comment, label, enqueue or merge.

Tip SHA: d5d667b

Fixes #697

The audit was correct and unusable: it failed on every bypass in the history that predates
merge-gate becoming a required check, so it was red on its first run and would stay red until
that history slid out of the window, and nothing ran it at all (#697).

--since takes a sha (resolved to its committer date) or an ISO date and starts the walk there:
a finding at or before the watermark prints as "earlier bypass" / "earlier direct commit" and
does not affect the exit code, so exit 1 now means a bypass newer than the watermark. With no
--since the behaviour is unchanged, which is what a hand run wants. A boundary commit counts as
earlier, and an unreadable --since is an error rather than a silent unfiltered walk.

The new read-only patrol (contents/actions/pull-requests: read) runs the script daily and takes
its watermark from the head_sha of its own previous successful run, so no file has to be
committed and the job cannot write anything. A first run has no previous run and baselines the
history it finds, which makes the first green the watermark.

The header comment is corrected too: it said a pass named the merge commit "was pushed to main",
which is more than the data shows, and it offered an `out` parameter that does not exist.

Fixes #697
@Birfy

Birfy commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Written-by: Dev

Closing this as a duplicate of #706, which was opened first and covers #697 (watermark plus the read-only patrol). One issue, one pull request — the older one stands; nothing here is needed from it.

For the record, so the difference is not lost: this branch also made --baseline and --since exclusive (giving both is an error instead of one silently winning), and it corrected the two comments noted on #694 (the "pushed to main" overstatement and the out parameter that does not exist). If either is wanted, it belongs in its own issue, not in a second PR for #697.

@Birfy Birfy closed this Oct 22, 2026
@Birfy Birfy removed the agent-pr Pull request opened by an agent label Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

audit-merges: report only bypasses newer than a watermark, and run it on a read-only patrol

1 participant