diff --git a/.github/workflows/pr-gate-test.yml b/.github/workflows/pr-gate-test.yml new file mode 100644 index 0000000..59eb3a1 --- /dev/null +++ b/.github/workflows/pr-gate-test.yml @@ -0,0 +1,23 @@ +name: pr-gate tests + +on: + pull_request: + paths: ['pr-gate/**', '.github/workflows/pr-gate*.yml'] + push: + branches: [main] + paths: ['pr-gate/**', '.github/workflows/pr-gate*.yml'] + +concurrency: + group: pr-gate-test-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '22' + - run: node --test pr-gate/decide.test.mjs diff --git a/.github/workflows/pr-gate.yml b/.github/workflows/pr-gate.yml new file mode 100644 index 0000000..df93a12 --- /dev/null +++ b/.github/workflows/pr-gate.yml @@ -0,0 +1,62 @@ +name: pr-gate + +# Reusable merge gate. Callers install pr-gate/caller.yml and pin this file to a commit SHA. +# It reads the PR through the API and never checks out or runs the PR's code, which is what makes +# running it from pull_request_target safe. +on: + workflow_call: + inputs: + pr-number: + description: PR to judge, as a string. Anything that is not a plain number is a no-op. + required: true + type: string + merge: + description: Squash-merge when every condition passes. False = report only. + required: false + type: boolean + default: false + gate-ref: + description: Commit of Starlight-Intelligence/.github to load the gate from. Pin it to the same SHA as the uses line. + required: false + type: string + default: main + required-checks: + description: Comma-separated check run names that must pass on the head commit, e.g. "test, build". + required: true + type: string + wait-minutes: + description: How long to wait for running CI when it is the only thing outstanding. + required: false + type: number + default: 8 + +jobs: + decide: + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: write + pull-requests: write + issues: write + checks: read + steps: + - name: Load the gate + uses: actions/checkout@v4 + with: + repository: Starlight-Intelligence/.github + ref: ${{ inputs.gate-ref }} + path: .pr-gate + persist-credentials: false + + - uses: actions/setup-node@v4 + with: + node-version: '22' + + - name: Decide + env: + GITHUB_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ inputs.pr-number }} + PR_GATE_MERGE: ${{ inputs.merge }} + PR_GATE_WAIT_MINUTES: ${{ inputs.wait-minutes }} + PR_GATE_REQUIRED_CHECKS: ${{ inputs.required-checks }} + run: node .pr-gate/pr-gate/run.mjs diff --git a/pr-gate/CODE-REVIEW-RULES.md b/pr-gate/CODE-REVIEW-RULES.md new file mode 100644 index 0000000..d66dda0 --- /dev/null +++ b/pr-gate/CODE-REVIEW-RULES.md @@ -0,0 +1,13 @@ +## Code Review Rules + +Codex reviews here as the second provider: most PRs in this repo are written by Claude. Flag as P0 or P1: + +- A claim in code, copy or docs with no evidence behind it: invented metrics, counts, testimonials, "verified" or "trusted by" without a source. +- Anything that fakes or bypasses CI: posting commit statuses by hand, skipping tests, `--admin`, `--no-verify`, tests rewritten to match broken behavior, graders a do-nothing run can pass. +- Secrets, tokens or keys in code, logs or tests; `.env` values committed. +- Money, auth, env or dependency changes, and new or deleted public pages, without a linked ruling from Frank. +- A checkout, buy button or install command for something that is not purchasable or published yet. Unlaunched products use the waitlist. +- Unbounded loops, retries or agent fan-out with no cap. +- On web surfaces: `100vh` in shells, hover styles outside `(hover: hover) and (pointer: fine)`, inputs under 16px, zoom disabled, motion with no `prefers-reduced-motion` path. + +Do not flag style preferences already enforced by the linter. diff --git a/pr-gate/README.md b/pr-gate/README.md new file mode 100644 index 0000000..d6a4d7b --- /dev/null +++ b/pr-gate/README.md @@ -0,0 +1,39 @@ +# pr-gate + +The estate merge gate, running on GitHub instead of a laptop. On the PR's current head commit it passes only when: + +- A review from a GitHub bot identity of a different harness than the author exists for that commit: `chatgpt-codex-connector[bot]` (Codex) or `claude[bot]` (Claude). The author is read from the branch prefix, because every harness pushes as the same GitHub account. +- That review is an approval, or a review whose body says it found no major issues. Dismissed and pending reviews are ignored. A bot review mentioning P0 or P1 blocks, and so does a bot comment that names a P0 or P1 against the head. Bot comments never sign off, because they carry no commit id. +- A renamed file is judged on both its old and new path. +- Every check named in the repo's `required-checks` has a latest attempt that succeeded, no other check failed, and nothing is still running. Commit statuses never count: anything holding a token can post one, which is how 20 PRs merged on 2026-10-19 with no check having run. +- All review threads are resolved (every page read; an unreadable answer blocks) and the full list of changed files was read. +- It is not a draft, carries no `hold` / `do-not-merge` / `blocked` / `wip` label, and its body links the intent it serves (an issue, board item or ruling). +- It touches none of Frank's paths: workflows, env files, dependencies, money, auth and middleware, `vercel.json`, or a new or deleted public page. No label overrides this. Labels prove nothing about who applied them on a shared account, so Frank merges those himself. + +In merge mode it decides again from fresh facts just before merging and merges with a head-SHA guard. + +## What does not count + +Typed lines (`starlight-signoff: ...` from the estate `tools/pr-gate.mjs`, `Checker: claude-cloud · head · verdict PASS`) are shown as advisory but never pass the gate: the author can type them. The estate `pr-gate.mjs` still honours them for a merge a human runs. + +So today, Claude-authored PRs pass once Codex reviews them. Codex- and Gemini-authored PRs pass only after a review from the Claude GitHub bot, or a human-run merge. + +Branch prefixes: `agent/claude/`, `claude/`, `ccr-` and `factory/` (Foreman, whose implementer is an Anthropic model) are Claude; `codex/`, `agent/codex/`, `agent/gpt*` are Codex; `agent/antigravity/`, `agent/gemini/`, Jules and `sentinel-` branches are Gemini. + +## Install in a repo + +1. Copy `caller.yml` to `.github/workflows/pr-gate.yml` and `poke.yml` to `.github/workflows/pr-gate-poke.yml`. Replace both `GATE_SHA` with a commit of this repo, list the repo's CI workflow names under `workflow_run`, and its check names in `required-checks`. +2. Turn on Codex code review and automatic reviews for the repo in Codex settings. +3. Add the section in `CODE-REVIEW-RULES.md` to the repo's `AGENTS.md`. +4. Start with `merge: false` (report only). Flip to `merge: true` where autonomous landing is ruled. + +Every trigger runs the caller from the default branch. Review events go through `pr-gate-poke`, which holds no permissions, and come back as `workflow_run`. + +## Known limits + +- Without branch protection nothing stops someone merging around the gate; the private repos on the free plan cannot have it. The gate is the merge path agents are told to use, not a lock. +- Between the final decision and the merge call a few milliseconds remain in which a review could be dismissed. Only a required status check under branch protection closes that. +- A same-repo PR's own workflows run with the repo token. The gate refuses to merge any PR that edits workflows, but cannot stop such a PR from posting a same-named check on itself. +- A squash merge made with the workflow token does not trigger push workflows on the base branch. Vercel's git integration still deploys. + +Tests: `node --test pr-gate/decide.test.mjs`. diff --git a/pr-gate/caller.yml b/pr-gate/caller.yml new file mode 100644 index 0000000..62f486d --- /dev/null +++ b/pr-gate/caller.yml @@ -0,0 +1,47 @@ +# Copy to .github/workflows/pr-gate.yml in a product repo, with poke.yml beside it. +# Replace GATE_SHA (both places) with a commit of Starlight-Intelligence/.github, list the repo's CI +# workflow names under workflow_run and its check names in required-checks, and set merge to true +# only where autonomous landing is ruled. +# +# Every trigger here runs this file from the default branch, never from the PR, so a PR cannot +# rewrite the gate it is judged by. Reviews arrive through pr-gate-poke (no permissions) and come +# back here as workflow_run. +name: pr-gate + +on: + pull_request_target: + types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled, edited] + issue_comment: + types: [created] + workflow_run: + workflows: [pr-gate-poke, CI] + types: [completed] + +permissions: + contents: write + pull-requests: write + issues: write + checks: read + +concurrency: + group: pr-gate-${{ github.event.pull_request.number || github.event.issue.number || github.event.workflow_run.pull_requests[0].number || github.event.workflow_run.display_title }} + cancel-in-progress: true + +jobs: + pr-gate: + # Comments re-run the gate only for a result from an allowlisted reviewer bot, or an exact + # `/gate` from an owner or org member. A comment can only trigger a re-read, never a sign-off. + if: >- + (github.event_name == 'pull_request_target') || + (github.event_name == 'workflow_run') || + (github.event_name == 'issue_comment' && github.event.issue.pull_request && ( + contains(fromJSON('["chatgpt-codex-connector[bot]","claude[bot]"]'), github.event.comment.user.login) || + (github.event.comment.body == '/gate' && contains(fromJSON('["OWNER","MEMBER"]'), github.event.comment.author_association)))) + uses: Starlight-Intelligence/.github/.github/workflows/pr-gate.yml@GATE_SHA + with: + # Reusable inputs are typed; the number is passed as a string. A relayed review with no PR + # attached falls back to the poke run title, which is the PR number. + pr-number: ${{ format('{0}', github.event.pull_request.number || github.event.issue.number || github.event.workflow_run.pull_requests[0].number || github.event.workflow_run.display_title) }} + required-checks: test + merge: false + gate-ref: GATE_SHA diff --git a/pr-gate/decide.mjs b/pr-gate/decide.mjs new file mode 100644 index 0000000..6a80dbd --- /dev/null +++ b/pr-gate/decide.mjs @@ -0,0 +1,187 @@ +// decide.mjs: the merge gate as a pure function over facts already fetched from GitHub. +// +// The local `tools/pr-gate.mjs` in the estate root enforced maker != checker, but only on a laptop, +// so nothing could land while it slept. This is the same rule moved onto GitHub, plus the checks the +// 2026-10-19 incident showed were missing: a forged commit status must never count as CI, and an +// unresolved review thread must never be merged over. + +// Every harness pushes as the same GitHub account, so the branch prefix is the only author signal. +// Keep in step with `tools/lib/harness-identity.mjs` in the estate root. +export function harnessOfBranch(branch) { + const b = String(branch || '').toLowerCase(); + if (b.startsWith('dependabot/') || b.startsWith('renovate/')) return 'dependabot'; + if (b.startsWith('agent/codex') || b.startsWith('codex/')) return 'codex'; + if (b.startsWith('agent/gpt')) return 'codex'; + if (b.startsWith('agent/claude') || b.startsWith('claude/') || b.startsWith('ccr-')) return 'claude'; + // Foreman (eve software factory) writes code with an Anthropic model; its in-pipeline reviewer + // does not count, because it saw the same plan. An outside non-Claude review is still required. + if (b.startsWith('factory/')) return 'claude'; + if (b.startsWith('agent/hermes')) return 'hermes'; + if (b.startsWith('agent/grok')) return 'grok'; + if (b.startsWith('agent/antigravity') || b.startsWith('agent/agy') || b.startsWith('agent/gemini')) return 'gemini'; + if (b.startsWith('agent/jules') || b.startsWith('sentinel-') || b.startsWith('jules/')) return 'gemini'; + if (b.startsWith('cursor/')) return 'cursor'; + if (b.startsWith('copilot/')) return 'copilot'; + if (b.startsWith('agent/')) return 'agent-other'; + return 'unlabeled'; +} + +// Under 7 hex chars a "sha" is a typo or a forgery and must never match a future head. +export function shaMatches(head, given) { + const h = String(head || '').toLowerCase(); + const g = String(given || '').toLowerCase(); + if (!/^[0-9a-f]{7,40}$/.test(h) || !/^[0-9a-f]{7,40}$/.test(g)) return false; + return g.length <= h.length ? h.startsWith(g) : g.startsWith(h); +} + +const MARKER_RE = /^\s*starlight-signoff:\s*harness=(\S+)\s+verdict=(\S+)\s+sha=([0-9a-fA-F]{7,40})\b/i; +const CHECKER_RE = /^\s*Checker:\s*([a-z0-9_-]+?)(?:-cloud)?\s*[·|-]\s*head\s+([0-9a-fA-F]{7,40})\s*[·|-]\s*verdict\s+(PASS|FAIL|BLOCK)\b/i; +const CODEX_CLEAN_RE = /didn'?t find any major issues/i; +const SEVERE_RE = /\bp[01]\b|p[01] badge/i; + +// GitHub's own bot accounts are the only reviewer identity an agent cannot type its way into. +// Exact logins on purpose: a substring match would accept any app whose name contains "codex". +const BOT_HARNESS = { 'chatgpt-codex-connector[bot]': 'codex', 'claude[bot]': 'claude' }; +const botHarness = (u) => (u && u.type === 'Bot' ? BOT_HARNESS[String(u.login || '').toLowerCase()] || null : null); + +// Turn comments and reviews into sign-offs. Typed lines (the estate `starlight-signoff` marker and +// `Checker:` lines) are collected but marked unauthenticated: every harness posts as the same human +// account, so anyone, the author included, can write one. They never count toward a pass here; the +// estate `tools/pr-gate.mjs` still honours them for a human-run merge. +export function collectSignoffs({ comments = [], reviews = [], headSha }) { + const head = String(headSha || '').toLowerCase(); + const out = []; + for (const c of comments) { + for (const line of String(c.body || '').split('\n')) { + let m = MARKER_RE.exec(line); + if (m) { out.push({ harness: m[1].toLowerCase(), verdict: m[2].toLowerCase(), sha: m[3].toLowerCase(), via: 'typed', trusted: false }); continue; } + m = CHECKER_RE.exec(line); + if (m) out.push({ harness: m[1].toLowerCase(), verdict: m[3].toLowerCase(), sha: m[2].toLowerCase(), via: 'typed', trusted: false }); + } + // Bot comments are not sign-offs: they carry no commit id, and a "no major issues" line can sit + // beside a P1 in the same comment. Only commit-bound bot reviews pass. A bot comment that names + // a P0/P1 against the head still blocks. + const h = botHarness(c.user); + if (h && SEVERE_RE.test(c.body || '') && head.length >= 7 && String(c.body).toLowerCase().includes(head.slice(0, 7))) { + out.push({ harness: h, verdict: 'block', sha: head, via: 'bot-comment', trusted: true }); + } + } + for (const r of reviews) { + const h = botHarness(r.user); + if (!h || r.state === 'DISMISSED' || r.state === 'PENDING') continue; + const body = String(r.body || ''); + let verdict = null; + if (r.state === 'CHANGES_REQUESTED' || SEVERE_RE.test(body)) verdict = 'block'; + else if (r.state === 'APPROVED' || CODEX_CLEAN_RE.test(body)) verdict = 'pass'; + // A COMMENTED review that neither approves nor says it is clean is not a verdict either way; + // its inline findings surface as unresolved threads. + if (verdict) out.push({ harness: h, verdict, sha: String(r.commitId || '').toLowerCase(), via: 'bot-review', trusted: true }); + } + return out; +} + +// Paths whose change is Frank's call under Doctrine 0 and AGENTS.md 5e: money, auth, env, +// dependencies, workflows (a PR that edits CI could edit this gate), and public route deletion. +const FRANK_PATHS = [ + [/^\.github\/workflows\//, 'edits CI workflows'], + [/(^|\/)\.env/, 'touches env files'], + [/(^|\/)(package\.json|pnpm-lock\.yaml|package-lock\.json|yarn\.lock|uv\.lock|pyproject\.toml)$/, 'changes dependencies'], + [/(checkout|billing|pricing|payment|payout|refund|stripe|polar|wallet)/i, 'touches a money path'], + [/(^|\/)(auth|middleware|proxy)(\.|\/)/i, 'touches auth or request middleware'], + [/(^|\/)vercel\.json$/, 'changes deploy config'], +]; +const PAGE_RE = /(^|\/)(app|pages)\/.*page\.(t|j)sx?$|(^|\/)pages\/(?!api\/).*\.(t|j)sx?$/; + +export function protectedReasons(files = []) { + const reasons = new Set(); + for (const f of files) { + // A rename is judged on both ends: moving a protected file out of its path still changes it. + for (const name of [f.filename, f.previousFilename].filter(Boolean)) { + for (const [re, why] of FRANK_PATHS) if (re.test(name)) reasons.add(`${why} (${name})`); + } + const renamed = f.status === 'renamed' && f.previousFilename; + if (f.status === 'removed' && PAGE_RE.test(f.filename)) reasons.add(`deletes a public page (${f.filename})`); + if (f.status === 'added' && PAGE_RE.test(f.filename)) reasons.add(`publishes a new public page (${f.filename})`); + if (renamed && PAGE_RE.test(f.previousFilename)) reasons.add(`deletes a public page (${f.previousFilename})`); + if (renamed && PAGE_RE.test(f.filename)) reasons.add(`publishes a new public page (${f.filename})`); + } + return [...reasons]; +} + +// CI counts only as check runs. Commit statuses are excluded on purpose: anything holding a token +// can post `state=success`, which is how 20 PRs merged on 2026-10-19 with no check having run. +// Each repo names the checks that must pass; one unrelated green check is not CI. +export function ciState(checkRuns = [], required = [], selfPrefix = 'pr-gate') { + const runs = checkRuns.filter((r) => !String(r.name || '').toLowerCase().startsWith(selfPrefix)); + const bad = runs.filter((r) => r.status === 'completed' && ['failure', 'cancelled', 'timed_out', 'action_required', 'stale', 'startup_failure'].includes(r.conclusion)); + const pending = runs.filter((r) => r.status !== 'completed'); + const missing = []; + const notPassed = []; + for (const name of required) { + const r = runs.find((x) => x.name === name); + if (!r) missing.push(name); + else if (r.status === 'completed' && r.conclusion !== 'success') notPassed.push(`${name} (${r.conclusion})`); + } + return { bad, pending, missing, notPassed }; +} + +const HOLD_LABELS = ['hold', 'do-not-merge', 'blocked', 'wip']; +const INTENT_RE = /(#\d+|github\.com\/[^\s/]+\/[^\s/]+\/issues\/\d+|claude\.ai\/artifact\/\S+)/i; + +export function decide(f) { + const author = harnessOfBranch(f.headRef); + const head = String(f.headSha || '').toLowerCase(); + const labels = (f.labels || []).map((l) => String(l).toLowerCase()); + const all = collectSignoffs({ comments: f.comments, reviews: f.reviews, headSha: head }); + const signoffs = all.filter((s) => s.trusted); + const onHead = (s) => shaMatches(head, s.sha); + + const reasons = []; + const waiting = []; + + if (f.state && f.state !== 'open') reasons.push(`PR is ${f.state}`); + if (f.draft) reasons.push('PR is a draft'); + const held = labels.filter((l) => HOLD_LABELS.includes(l)); + if (held.length) reasons.push(`held by label: ${held.join(', ')}`); + + if (!INTENT_RE.test(f.body || '')) { + reasons.push('no linked intent: reference the issue, board item or ruling this PR serves'); + } + + const blocks = signoffs.filter((s) => (s.verdict === 'block' || s.verdict === 'fail') && onHead(s) && s.harness !== author); + const valid = signoffs.filter((s) => s.verdict === 'pass' && onHead(s) && s.harness !== author); + if (blocks.length) reasons.push(`blocked on this commit by ${[...new Set(blocks.map((b) => b.harness))].join(', ')}`); + if (!valid.length) { + const stale = signoffs.filter((s) => s.verdict === 'pass' && s.harness !== author && !onHead(s)); + if (stale.length) waiting.push(`cross-harness review exists for an older commit; head is ${head.slice(0, 7)}`); + else waiting.push(`needs a review from a harness other than ${author} on ${head.slice(0, 7)}`); + } + + if (f.incomplete) reasons.push(`could not read the whole PR: ${f.incomplete}`); + if ((f.unresolvedThreads || 0) > 0) reasons.push(`${f.unresolvedThreads} unresolved review thread(s)`); + + const required = f.requiredChecks || []; + const ci = ciState(f.checkRuns, required, f.selfPrefix); + if (!required.length) reasons.push('no required CI checks are configured for this repo'); + if (ci.bad.length) reasons.push(`CI failing: ${ci.bad.map((r) => r.name).join(', ')}`); + if (ci.notPassed.length) reasons.push(`required check did not pass: ${ci.notPassed.join(', ')}`); + if (ci.pending.length) waiting.push(`CI still running: ${ci.pending.map((r) => r.name).join(', ')}`); + else if (ci.missing.length) reasons.push(`required check never ran on this commit: ${ci.missing.join(', ')} (commit statuses do not count)`); + + // No label can override this: every harness acts as the same account, so a label proves nothing + // about who applied it. Protected changes are merged by Frank, outside this gate. + const prot = protectedReasons(f.files); + if (prot.length) reasons.push(...prot.map((p) => `Frank merges this one: ${p}`)); + + const typed = all.filter((s) => !s.trusted && s.verdict === 'pass' && onHead(s) && s.harness !== author); + return { + ok: reasons.length === 0 && waiting.length === 0, + pending: reasons.length === 0 && waiting.length > 0, + reasons, + waiting, + author, + head, + signers: [...new Set(valid.map((s) => s.harness))], + advisory: [...new Set(typed.map((s) => s.harness))], + }; +} diff --git a/pr-gate/decide.test.mjs b/pr-gate/decide.test.mjs new file mode 100644 index 0000000..a440d77 --- /dev/null +++ b/pr-gate/decide.test.mjs @@ -0,0 +1,138 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { decide, harnessOfBranch, shaMatches, protectedReasons, ciState } from './decide.mjs'; + +const HEAD = 'a1b2c3d4e5f60718293a4b5c6d7e8f9012345678'; +const codexBot = { login: 'chatgpt-codex-connector[bot]', type: 'Bot' }; +const claudeBot = { login: 'claude[bot]', type: 'Bot' }; +const human = { login: 'frankxai', type: 'User' }; +const greenCI = [{ name: 'test', status: 'completed', conclusion: 'success' }]; +const cleanReview = { user: codexBot, commitId: HEAD, state: 'COMMENTED', body: "Codex Review: Didn't find any major issues." }; + +const base = (over = {}) => ({ + headRef: 'agent/claude/thing', + headSha: HEAD, + draft: false, + state: 'open', + body: 'Serves #12', + labels: [], + files: [{ filename: 'src/lib/x.ts', status: 'modified' }], + comments: [], + reviews: [cleanReview], + checkRuns: greenCI, + requiredChecks: ['test'], + unresolvedThreads: 0, + ...over, +}); + +test('clean Claude PR with a clean Codex bot review on head passes', () => { + const d = decide(base()); + assert.equal(d.ok, true, JSON.stringify(d)); + assert.deepEqual(d.signers, ['codex']); +}); + +test('author cannot be the checker', () => { + const d = decide(base({ headRef: 'codex/x' })); + assert.equal(d.ok, false); + assert.match(d.waiting.join(), /other than codex/); +}); + +test('a review of an older commit does not count', () => { + const d = decide(base({ reviews: [{ ...cleanReview, commitId: 'ffffffffffffffffffffffffffffffffffffffff' }] })); + assert.equal(d.ok, false); + assert.match(d.waiting.join(), /older commit/); +}); + +// Codex review of b526896 reproduced these three as false passes. +test('typed sign-off lines are unauthenticated and never pass, even with the right sha', () => { + for (const body of [`starlight-signoff: harness=claude verdict=pass sha=${HEAD}`, `Checker: claude-cloud · head ${HEAD} · verdict PASS`]) { + const d = decide(base({ headRef: 'codex/x', reviews: [], comments: [{ user: human, body }] })); + assert.equal(d.ok, false, body); + assert.ok(d.advisory.length > 0, 'still reported as advisory'); + } +}); + +test('dismissed and pending bot reviews do not count', () => { + for (const state of ['DISMISSED', 'PENDING']) assert.equal(decide(base({ reviews: [{ ...cleanReview, state }] })).ok, false, state); +}); + +test('a bot review that is neither approval nor clean is not a pass; P0/P1 text blocks', () => { + assert.equal(decide(base({ reviews: [{ user: codexBot, commitId: HEAD, state: 'COMMENTED', body: 'Critical security issue' }] })).ok, false); + const d = decide(base({ reviews: [{ user: codexBot, commitId: HEAD, state: 'COMMENTED', body: '![P1 Badge] token leaks' }] })); + assert.match(d.reasons.join(), /blocked on this commit by codex/); +}); + +test('a human account or a look-alike bot claiming to be Codex signs nothing', () => { + assert.equal(decide(base({ reviews: [{ ...cleanReview, user: human }] })).ok, false); + assert.equal(decide(base({ reviews: [{ ...cleanReview, user: { login: 'my-codex-helper[bot]', type: 'Bot' } }] })).ok, false); +}); + +// Codex review of 1404c37 reproduced the next two as false passes. +test('bot comments never sign off; a P0/P1 bot comment on the head blocks', () => { + const clean = decide(base({ reviews: [], comments: [{ user: codexBot, body: `Didn't find any major issues. Reviewed ${HEAD.slice(0, 10)}` }] })); + assert.equal(clean.ok, false); + const mixed = decide(base({ comments: [{ user: codexBot, body: `Didnt find any major issues on ${HEAD.slice(0, 7)}. P1: auth bypass remains.` }] })); + assert.match(mixed.reasons.join(), /blocked on this commit by codex/); + assert.equal(decide(base({ reviews: [{ user: codexBot, commitId: HEAD, state: 'COMMENTED', body: 'p1 regression' }] })).ok, false); +}); + +test('renames are judged on both ends', () => { + assert.ok(protectedReasons([{ filename: 'docs/old-ci.yml', previousFilename: '.github/workflows/ci.yml', status: 'renamed' }]).length > 0); + const page = protectedReasons([{ filename: 'app/new/page.tsx', previousFilename: 'app/about/page.tsx', status: 'renamed' }]); + assert.equal(page.length, 2); + assert.equal(decide(base({ files: [{ filename: 'app/new/page.tsx', previousFilename: 'app/about/page.tsx', status: 'renamed' }] })).ok, false); +}); + +test('Claude bot approval signs a Codex PR on the exact head', () => { + const d = decide(base({ headRef: 'codex/x', reviews: [{ user: claudeBot, commitId: HEAD, state: 'APPROVED', body: '' }] })); + assert.equal(d.ok, true, JSON.stringify(d)); + assert.equal(shaMatches(HEAD, 'a1b2'), false); +}); + +test('required checks must exist and pass; skipped or unrelated green is not CI', () => { + assert.match(decide(base({ checkRuns: [] })).reasons.join(), /never ran/); + assert.match(decide(base({ checkRuns: [{ name: 'docs', status: 'completed', conclusion: 'success' }, { name: 'test', status: 'completed', conclusion: 'skipped' }] })).reasons.join(), /did not pass: test \(skipped\)/); + assert.match(decide(base({ requiredChecks: [] })).reasons.join(), /no required CI checks/); +}); + +test('failing or running CI blocks or waits; the gate ignores its own runs', () => { + assert.match(decide(base({ checkRuns: [...greenCI, { name: 'Lint', status: 'completed', conclusion: 'failure' }] })).reasons.join(), /CI failing: Lint/); + const w = decide(base({ checkRuns: [...greenCI, { name: 'build', status: 'in_progress', conclusion: null }] })); + assert.equal(w.pending, true); + assert.equal(ciState([...greenCI, { name: 'pr-gate / decide', status: 'completed', conclusion: 'failure' }], ['test']).bad.length, 0); +}); + +test('unresolved threads, incomplete reads, drafts, hold labels and missing intent all block', () => { + assert.equal(decide(base({ unresolvedThreads: 2 })).ok, false); + assert.match(decide(base({ incomplete: 'saw 3000 of 3200 changed files' })).reasons.join(), /could not read/); + assert.equal(decide(base({ draft: true })).ok, false); + assert.equal(decide(base({ labels: ['hold'] })).ok, false); + assert.match(decide(base({ body: 'no link here' })).reasons.join(), /no linked intent/); +}); + +test('changes requested by the Codex bot block', () => { + const d = decide(base({ reviews: [{ user: codexBot, commitId: HEAD, state: 'CHANGES_REQUESTED', body: '' }] })); + assert.match(d.reasons.join(), /blocked on this commit by codex/); +}); + +test("Frank's paths block and no label overrides them", () => { + for (const filename of ['.github/workflows/ci.yml', 'package.json', 'app/checkout/route.ts', '.env.production', 'src/middleware.ts', 'vercel.json']) { + const d = decide(base({ files: [{ filename, status: 'modified' }], labels: ['frank-approved'] })); + assert.equal(d.ok, false, filename); + assert.match(d.reasons.join(), /Frank merges this one/, filename); + } +}); + +test('publishing or deleting a public page is protected; editing one is not', () => { + assert.equal(protectedReasons([{ filename: 'app/playbooks/page.tsx', status: 'added' }]).length, 1); + assert.equal(protectedReasons([{ filename: 'app/about/page.tsx', status: 'removed' }]).length, 1); + assert.equal(protectedReasons([{ filename: 'app/about/page.tsx', status: 'modified' }]).length, 0); +}); + +test('branch prefixes map to harnesses, Foreman and cloud sessions count as Claude', () => { + assert.equal(harnessOfBranch('factory/fix-reset-email'), 'claude'); + assert.equal(harnessOfBranch('ccr-a491b337-99lzm2'), 'claude'); + assert.equal(harnessOfBranch('agent/gpt6/motion'), 'codex'); + assert.equal(harnessOfBranch('sentinel-fix-timing'), 'gemini'); + assert.equal(harnessOfBranch('dependabot/npm_and_yarn/x'), 'dependabot'); +}); diff --git a/pr-gate/poke.yml b/pr-gate/poke.yml new file mode 100644 index 0000000..8f92ef2 --- /dev/null +++ b/pr-gate/poke.yml @@ -0,0 +1,20 @@ +# Copy to .github/workflows/pr-gate-poke.yml. It relays review events to pr-gate. +# pull_request_review runs workflow code from the PR itself, so this file holds no permissions and +# does nothing but finish; pr-gate picks it up via workflow_run from the default branch. +name: pr-gate-poke +# The run title carries the PR number back to pr-gate, which re-reads that PR from the API, so a +# forged title can only cause a re-read of some PR, never a decision on fabricated facts. +run-name: '${{ github.event.pull_request.number }}' + +on: + pull_request_review: + types: [submitted, dismissed] + +permissions: {} + +jobs: + poke: + runs-on: ubuntu-latest + timeout-minutes: 1 + steps: + - run: echo "review on PR ${{ github.event.pull_request.number }}" diff --git a/pr-gate/run.mjs b/pr-gate/run.mjs new file mode 100644 index 0000000..9f74f08 --- /dev/null +++ b/pr-gate/run.mjs @@ -0,0 +1,143 @@ +// run.mjs: fetch the facts for one PR, decide, report, and merge on a clean pass. +// No dependencies: Node 20+ fetch against the GitHub API with the workflow's GITHUB_TOKEN. +// It never checks out or executes the PR's code. +import { appendFileSync } from 'node:fs'; +import { decide } from './decide.mjs'; + +const token = process.env.GITHUB_TOKEN; +const repo = process.env.GITHUB_REPOSITORY; +const rawNumber = String(process.env.PR_NUMBER || '').trim(); +const merge = process.env.PR_GATE_MERGE === 'true'; +const waitMinutes = Number(process.env.PR_GATE_WAIT_MINUTES || 8); +const selfPrefix = (process.env.PR_GATE_SELF_PREFIX || 'pr-gate').toLowerCase(); +const requiredChecks = String(process.env.PR_GATE_REQUIRED_CHECKS || '').split(',').map((s) => s.trim()).filter(Boolean); +const STICKY = ''; + +if (!token || !repo) { console.error('GITHUB_TOKEN and GITHUB_REPOSITORY are required'); process.exit(2); } +// A CI run with no PR attached passes its own title here; that is not a PR to judge. +if (!/^[1-9][0-9]{0,8}$/.test(rawNumber)) { console.log(`no PR number in "${rawNumber.slice(0, 80)}"; nothing to judge`); process.exit(0); } +const number = Number(rawNumber); +const [owner, name] = repo.split('/'); + +async function api(path, init = {}) { + const res = await fetch(`https://api.github.com${path}`, { + ...init, + headers: { authorization: `Bearer ${token}`, accept: 'application/vnd.github+json', 'x-github-api-version': '2022-11-28', ...(init.headers || {}) }, + }); + if (!res.ok) throw Object.assign(new Error(`${init.method || 'GET'} ${path} -> ${res.status} ${await res.text()}`), { status: res.status }); + return res.status === 204 ? null : res.json(); +} +// Fails closed: running out of pages before running out of items is an error, never a short list. +async function all(path) { + const out = []; + for (let page = 1; page <= 50; page++) { + const sep = path.includes('?') ? '&' : '?'; + const batch = await api(`${path}${sep}per_page=100&page=${page}`); + const items = Array.isArray(batch) ? batch : batch.check_runs; + out.push(...items); + if (items.length < 100) return out; + } + throw new Error(`${path} has more than 5,000 items`); +} +// Every page, and a GraphQL error is a failure: a missing answer must never read as zero threads. +async function unresolvedThreads() { + const q = `query($o:String!,$n:String!,$p:Int!,$c:String){repository(owner:$o,name:$n){pullRequest(number:$p){reviewThreads(first:100,after:$c){nodes{isResolved}pageInfo{hasNextPage endCursor}}}}}`; + let count = 0; + let cursor = null; + for (let i = 0; i < 50; i++) { + const r = await api('/graphql', { method: 'POST', body: JSON.stringify({ query: q, variables: { o: owner, n: name, p: number, c: cursor } }) }); + const t = r.data?.repository?.pullRequest?.reviewThreads; + if (r.errors?.length || !t) throw new Error(`review threads unreadable: ${JSON.stringify(r.errors || 'no data').slice(0, 300)}`); + count += t.nodes.filter((n) => !n.isResolved).length; + if (!t.pageInfo.hasNextPage) return count; + cursor = t.pageInfo.endCursor; + } + throw new Error('review threads exceed 5,000'); +} + +async function facts() { + const pr = await api(`/repos/${repo}/pulls/${number}`); + const head = pr.head.sha; + const [files, comments, reviews, checkRuns, threads] = await Promise.all([ + all(`/repos/${repo}/pulls/${number}/files`), + all(`/repos/${repo}/issues/${number}/comments`), + all(`/repos/${repo}/pulls/${number}/reviews`), + // filter=latest: only the newest attempt of each check counts, so a rerun that failed wins + // over an older success. + all(`/repos/${repo}/commits/${head}/check-runs?filter=latest`), + unresolvedThreads(), + ]); + // The files endpoint stops at 3,000; a protected path past that point must not go unseen. + const incomplete = files.length < pr.changed_files ? `saw ${files.length} of ${pr.changed_files} changed files` : ''; + return { + pr, + f: { + headRef: pr.head.ref, + headSha: head, + incomplete, + requiredChecks, + draft: pr.draft, + state: pr.state, + body: pr.body || '', + labels: pr.labels.map((l) => l.name), + files: files.map((x) => ({ filename: x.filename, status: x.status, previousFilename: x.previous_filename })), + comments: comments.filter((c) => !String(c.body || '').includes(STICKY)).map((c) => ({ body: c.body, user: c.user, createdAt: c.created_at })), + reviews: reviews.map((r) => ({ user: r.user, commitId: r.commit_id, state: r.state, body: r.body })), + checkRuns: checkRuns.map((r) => ({ name: r.name, status: r.status, conclusion: r.conclusion })), + unresolvedThreads: threads, + selfPrefix, + }, + }; +} + +// Wait only when CI is the sole thing outstanding, so a PR that is otherwise clean lands in the +// same run instead of waiting for the next event. +let { pr, f } = await facts(); +let d = decide(f); +const deadline = Date.now() + waitMinutes * 60_000; +while (d.pending && d.waiting.every((w) => w.startsWith('CI still running')) && Date.now() < deadline) { + await new Promise((r) => setTimeout(r, 30_000)); + ({ pr, f } = await facts()); + d = decide(f); +} + +let merged = null; +if (d.ok && merge) { + // Decide again from fresh facts immediately before merging, so a dismissal, hold label or failed + // rerun during the wait is seen. The window that remains is milliseconds, not minutes; only a + // required check under branch protection closes it fully. + ({ pr, f } = await facts()); + d = decide(f); +} +if (d.ok && merge) { + try { + // `sha` makes the merge refuse if the head moved after the decision was made. + const r = await api(`/repos/${repo}/pulls/${number}/merge`, { method: 'PUT', body: JSON.stringify({ merge_method: 'squash', sha: d.head }) }); + merged = r.sha; + } catch (e) { + d.ok = false; + d.reasons.push(`merge refused by GitHub: ${e.message.slice(0, 300)}`); + } +} + +const verdict = merged ? 'MERGED' : d.ok ? 'PASS' : d.pending ? 'WAITING' : 'BLOCKED'; +const lines = [ + STICKY, + `**pr-gate: ${verdict}** on \`${d.head.slice(0, 7)}\` · author harness \`${d.author}\`${d.signers.length ? ` · reviewed by ${d.signers.join(', ')}` : ''}`, + d.advisory.length ? `- typed sign-off from ${d.advisory.join(', ')} noted; typed lines are unauthenticated and do not count here (use the estate pr-gate for a human-run merge)` : '', + ...d.reasons.map((r) => `- blocked: ${r}`), + ...d.waiting.map((w) => `- waiting: ${w}`), + merged ? `- squash-merged as \`${merged.slice(0, 7)}\`` : '', + d.ok && !merge ? '- this repo runs the gate in report-only mode; merge through `pr-gate merge`' : '', + '', + '_Re-evaluates on push, bot review, bot comment, label and CI completion. Comment `/gate` to re-run. A new push invalidates every review._', +].filter(Boolean); +const body = lines.join('\n'); + +const existing = (await all(`/repos/${repo}/issues/${number}/comments`)).find((c) => String(c.body || '').includes(STICKY)); +if (existing) await api(`/repos/${repo}/issues/comments/${existing.id}`, { method: 'PATCH', body: JSON.stringify({ body }) }); +else await api(`/repos/${repo}/issues/${number}/comments`, { method: 'POST', body: JSON.stringify({ body }) }); + +if (process.env.GITHUB_STEP_SUMMARY) appendFileSync(process.env.GITHUB_STEP_SUMMARY, `${body}\n`); +console.log(body); +process.exit(d.ok ? 0 : 1);