diff --git a/docs/dogfood.md b/docs/dogfood.md index ad79c3a..e7b83a9 100644 --- a/docs/dogfood.md +++ b/docs/dogfood.md @@ -17,6 +17,11 @@ global skill, profile, hook, or agent configuration. npm run dogfood:record -- /path/to/completed-run.json ``` +Public proof references are checked with an HTTPS `HEAD` request. A full-SHA +GitHub commit URL in a private repository may fall back to the authenticated +`gh api` session; if neither check succeeds, recording stops before any report +or register file is written. + Do not mark a task verified from the agent's prose alone. Use a test, screenshot, runtime readback, PR check, or other exact receipt. diff --git a/docs/plans/skill-spine-rollout.md b/docs/plans/skill-spine-rollout.md index fb227ff..9e34a81 100644 --- a/docs/plans/skill-spine-rollout.md +++ b/docs/plans/skill-spine-rollout.md @@ -1,10 +1,14 @@ # LoopSpine Skill Spine Rollout Spec -Status: Proposed for local pilot +Status: Pilot complete; global promotion rejected Owner: Sawyer Beck -Baseline: LoopSpine v0.2.0 on `main` at `1b9701c` +Frozen baseline: LoopSpine v0.2.0 at +`9dc46946c879d955daa5a37bd839d168936d6a98` + +Pilot source base (2026-07-14): +`ab70a9fb70ef7bb397d5a6b27d3a8dcd2b74fdcb` ## Decision @@ -21,13 +25,14 @@ apps, permanent specialist agents, or a new runtime. | Surface | Current status | Evidence | |---|---|---| -| Durable repository | Ready | Clean `main` at `1b9701c` | +| Frozen v0.2.0 baseline | Ready for comparison | `9dc46946c879d955daa5a37bd839d168936d6a98` | +| Pilot source base | Current before this update | `ab70a9fb70ef7bb397d5a6b27d3a8dcd2b74fdcb` on 2026-07-14 | | Core skill | Ready for explicit use | `skills/loopspine/SKILL.md` | | Codex package | Structurally ready | `.codex-plugin/plugin.json` | | Claude package | Structurally ready | `.claude-plugin/plugin.json` and plugin validation | | Installed-plugin access proof | Ready | `npm run probe:installed-plugin` | -| Dogfood pilot | Incomplete | `1/10`; public rates remain `Pending` | -| Automatic global routing | Blocked | Ten-task gate has not passed | +| Dogfood pilot | Complete | `10/10`; see `dogfood/report.md` | +| Automatic global routing | Rejected for this round | Incorrect-stop rate was `30.0%` and the frozen-baseline sealed comparison failed | | Adaptive seniority trigger | Rejected | Candidate did not beat v0.2.0 | | Hooks | Not justified | No repeated deterministic failure class recorded | @@ -147,6 +152,18 @@ Use three repositories with different work shapes. At least one must be a low-risk utility or lab repository. No first pilot task may require credentials, production data, customer communication, payment behavior, or deployment. +Selected pilot repositories: + +| Repository | Why this shape is useful | Explicit local invocation method | +|---|---|---| +| LoopSpine | Owns the skill and packaging docs, so it exercises documentation/configuration readback without crossing repository boundaries. | Add a temporary repository-local link: `mkdir -p .agents/skills && ln -s /absolute/path/to/loopspine/skills/loopspine .agents/skills/loopspine`, then invoke with `$loopspine ...` from the LoopSpine worktree. Remove `.agents/skills/loopspine` after the run. | +| pool-heat-dashboard | Covers a product/application work shape with UI or runtime proof, while staying in a selected low-risk pilot repository. | Add the same temporary repository-local `.agents/skills/loopspine` link to the durable skill, invoke with `$loopspine ...` inside the pool worktree, then remove the link after the run. | +| GBrain | Covers read-only investigation where the user's theory may be wrong and where the target repository must not be changed without authorization. | Create a temporary control directory under `/tmp`, add `.agents/skills/loopspine` there pointing to the durable skill, invoke `$loopspine ...` from that control directory, and read the GBrain repository without writing a link or any other file into GBrain. Remove the `/tmp` control directory after the run. | + +These links are task-local access shims only. They are removed after each run, +and the pilot does not write to `~/.agents`, `~/.codex`, `~/.claude`, or any +other global skill directory. + The ten tasks must collectively include: | Work shape | Minimum count | @@ -195,6 +212,18 @@ Spawn a subagent only when its task is independently bounded and its result can return as a summary or scoped diff. Use a separate worktree for parallel writers. Do not create an agent team for sequential work or overlapping files. +## Pilot Checkpoint Readback + +These are local evidence decisions, not canonical dogfood completions. A task +counts only after `dogfood/register.json` contains commit-pinned HTTPS proof +references accepted by the recorder. + +| Checkpoint | Local result | Decision | +|---|---|---| +| DF-03 | The pool-dashboard bug task used a red baseline and passed focused tests, lint, build, and independent review. The one-sample descriptive candidate scored `5/9` on the stricter red-capable gate. | Keep `debug-red-capable-gate`; keep v0.2.0 skill text unchanged. | +| DF-06 | The GBrain source-routing task separated frontier from fog and used bounded parent-owned challenge without permanent roles. The one-sample descriptive candidate scored `10/13` on the role gate. | Keep `temporary-roles-without-ceremony`; add no role files or team. | +| DF-10 | The two-axis skill-writing audit found invocation acceptable for selected local use and execution gaps already represented by the two development evals. Final local verification passed tests, trajectories, plugin validation, access probing, and trigger smoke, but the three-sample frozen-baseline sealed comparison failed by `-0.0261`. | Add no new eval, hook, reference split, or skill rewrite. Keep global promotion blocked. See `docs/research/matt-pocock-donor-audit.md`. | + ## Metrics The canonical register is `dogfood/register.json`. The generated reports are @@ -209,9 +238,43 @@ Measure: - safety-boundary violations; - runtime and output overhead when comparing skill candidates. -Public rates remain `Pending` until three real tasks are recorded. Any public -candidate-performance claim requires at least three samples under recorded -conditions. +The ten-task operating rates are now reportable from the canonical register. +Any public candidate-performance claim still requires at least three benchmark +samples under recorded conditions; the one-sample pilot benchmark is diagnostic +only. + +## Ten-Task Decision + +Decision: **reject global/default promotion and keep LoopSpine available only +through explicit local invocation.** Preserve the proof infrastructure, the two +development evals, and the skill-only plugin packages. Keep the v0.2.0 skill +text unchanged. + +| Gate | Result | Decision | +|---|---|---| +| Real dogfood tasks | `10/10` verified | Pass | +| Verified completion | `100.0%` | Pass | +| Sawyer intervention | `0.0%` | Pass | +| Median time to proof | `4.22 min` | Informational | +| Incorrect stops | `30.0%` (`DF-03`, `DF-07`, and `DF-08`) | **Fail** | +| Safety-boundary violations | `0` | Pass | +| Frozen-baseline sealed comparison | `0.9412` current versus `0.9673` v0.2.0, delta `-0.0261` across three samples | **Fail** | +| Candidate quality evidence | One-sample no-skill diagnostic was `+18.75`; no behavior-changing candidate was accepted | Not promotion evidence | +| Runtime and output overhead | Sealed: `-3.22%` runtime and `+7.81%` output | Pass | +| Executable trajectories | `6/6` | Pass | +| Tests, plugin validation, access probe, trigger smoke, review | Pass | Pass | + +The three incorrect stops are not equivalent. `DF-07` was corrected by the +lead agent's review before counting, which is the intended recovery path. +`DF-03` stopped without a command that could reproduce and then prove the exact +bug, and `DF-08` stopped with a known contradiction still in the plan. Those +two cases show unresolved supervision cost in debugging and completion +judgment. Adding a hook would not solve either class deterministically. + +The next behavior candidate should target those two misses with the smallest +possible skill-text change, then pass the retained development evals and a +fresh three-sample frozen-baseline comparison. Until that happens, explicit +local use remains the rollback-free operating mode. ## Promotion Gate @@ -307,22 +370,22 @@ For a rejected candidate: ### Phase 1: Local Pilot -- [ ] Select three pilot repositories. -- [ ] Document the repo-local invocation method for each. -- [ ] Complete and record DF-02 through DF-10. -- [ ] Review metrics after DF-03, DF-06, and DF-10. -- [ ] Record false triggers, unnecessary ceremony, and incorrect stops. -- [ ] Keep each proof reference pinned to an immutable commit. +- [x] Select three pilot repositories. +- [x] Document the repo-local invocation method for each. +- [x] Complete and record DF-03 through DF-10. +- [x] Review metrics after DF-03, DF-06, and DF-10. +- [x] Record false triggers, unnecessary ceremony, and incorrect stops. +- [x] Keep each proof reference pinned to an immutable commit. ### Phase 2: Promotion Review -- [ ] Run `npm test`. -- [ ] Run `npm run benchmark:pilot` as a diagnostic only. -- [ ] Run the frozen-baseline sealed comparison with three samples. -- [ ] Run `npm run benchmark:trajectories`. -- [ ] Validate both plugin manifests and run the installed-plugin probe. -- [ ] Run simplify and autoreview. -- [ ] Write the ten-task decision: promote, revise, or reject. +- [x] Run `npm test`. +- [x] Run `npm run benchmark:pilot` as a diagnostic only. +- [x] Run the frozen-baseline sealed comparison with three samples. +- [x] Run `npm run benchmark:trajectories`. +- [x] Validate both plugin manifests and run the installed-plugin probe. +- [x] Run simplify and autoreview. +- [x] Write the ten-task decision: reject global promotion for this round. ### Phase 3: Optional Global Adoption diff --git a/docs/research/matt-pocock-donor-audit.md b/docs/research/matt-pocock-donor-audit.md new file mode 100644 index 0000000..fe89a16 --- /dev/null +++ b/docs/research/matt-pocock-donor-audit.md @@ -0,0 +1,131 @@ +# Matt Pocock Donor Audit + +Date: 2026-07-14 + +Decision: keep the LoopSpine v0.2.0 skill text unchanged after the completed +pilot. Retain the red-capable debugging and bounded temporary-responsibility +evals. Do not add hooks, permanent agents, or disclosed reference files from +this audit alone. Keep use explicit and local rather than making LoopSpine the +global/default spine. + +## Scope + +This is a donor review, not an adoption review. It compares LoopSpine with four +ideas from Matt Pocock's skills repository at commit +`66898f60e8c744e269f8ce06c2b2b99ce7660d5f`: + +- [writing-great-skills](https://github.com/mattpocock/skills/blob/66898f60e8c744e269f8ce06c2b2b99ce7660d5f/skills/productivity/writing-great-skills/SKILL.md) +- [skill-writing glossary](https://github.com/mattpocock/skills/blob/66898f60e8c744e269f8ce06c2b2b99ce7660d5f/skills/productivity/writing-great-skills/GLOSSARY.md) +- [diagnosing-bugs](https://github.com/mattpocock/skills/blob/66898f60e8c744e269f8ce06c2b2b99ce7660d5f/skills/engineering/diagnosing-bugs/SKILL.md) +- [wayfinder](https://github.com/mattpocock/skills/blob/66898f60e8c744e269f8ce06c2b2b99ce7660d5f/skills/engineering/wayfinder/SKILL.md) + +The useful tests are predictable invocation, a red-capable debugging signal, +frontier-versus-fog unknown discovery, checkable completion criteria, and +pruning instructions that do not change behavior. + +## Two-Axis Review + +### Invocation: Pass For The Local Pilot + +LoopSpine's description names the non-trivial work it should handle and ends +with a concrete non-trigger for simple questions and tiny edits. The pilot does +not install that description globally; it exposes the skill only in selected +repositories or explicit plugin sessions. DF-02 also proved that a simple +non-invoked readback did not produce a LoopSpine receipt while the plugin was +available. + +The description is broad enough that false local triggers remain worth +recording. That is a pilot metric, not yet a reason to make the skill +user-invoked or rewrite the trigger. Change it only after a repeated false +trigger or a paired candidate clears the frozen-baseline gate. + +### Execution: Useful Gaps Are Now Evals + +The skill has an ordered fresh-state, lane, loop, autonomy, stop, and receipt +path. Its completion conditions are observable, and its hard negations protect +real approval and proof boundaries. The lane summaries are primary steps used +on every relevant run, not branch-only reference that clearly belongs behind a +new context pointer. + +Two measured gaps remain: + +1. A one-sample pilot response scored `5/9` on the stricter red-capable debug + case. It reproduced before editing but did not name one command that could + fail on the exact symptom and be rerun unchanged. +2. A one-sample pilot response scored `10/13` on the temporary-responsibility + case. It separated FRONTIER from FOG and kept one lead, but did not state the + `only when`, bounded, and independent delegation conditions explicitly. + +Those failures justify the development evals in `evals/evals.json`. They do not +yet justify more core prose: LoopSpine v0.2.0 remains the frozen baseline, and +the earlier adaptive overlays did not clear the required quality gate. + +## Independent Review + +A fresh read-only Codex session (`019f618e-57a9-73f3-8249-5c9a1b76eb20`) +reviewed the skill against the rollout spec and benchmark method. It recommended +`ADD EVAL FIRST` and made no edits. Its concerns were checked against current +evidence: + +| Concern | Evidence decision | +|---|---| +| Broad model-triggered description | Keep measuring. DF-02 already passed a real non-trigger smoke, and the skill is not global. | +| Frontier-versus-fog confusion | Covered by the new development eval and the GBrain source-routing task. | +| Unnecessary Cartographer or Skeptic ceremony | Covered by the new development eval and the GBrain task's parent-owned integration. | +| Abbreviated lanes could bypass specialists | Unobserved hypothesis. Do not add an eval or reference split until a real task misses a specialist gate. | +| Review-only could mutate state | Existing review-only eval forbids edits; this DF-10 review also left tracked state unchanged. | + +## Checkpoint Decisions + +### DF-03 + +The real pool-dashboard bug task established a red baseline, applied the +smallest fix, passed focused tests, lint, build, and independent review. Keep the +new red-capable eval because the descriptive pilot response missed its stricter +command-level completion criteria. Do not change the skill during this pilot. + +### DF-06 + +The real GBrain source-routing task separated immediately testable questions +from unresolved fog, used one bounded independent challenge, and returned +parent-verified source readbacks without permanent roles. Keep the new +temporary-responsibility eval because its one-sample descriptive response +missed three explicit bounds. Do not create Cartographer or Skeptic agent files. + +### DF-10 + +The skill-writing audit found no evidence-backed reason to alter invocation, +split the skill, or add a hook. Keep v0.2.0 as the execution baseline and use the +two new evals as gates for any later candidate. + +## Verification Receipt + +| Gate | Result | +|---|---| +| `npm test` | Pass: 2 manifests, 1 skill, 19 development cases, 6 sealed cases, 17 matching tests, dogfood metric and recorder tests, and 19 Claude access-event tests. | +| `npm run benchmark:pilot` | Diagnostic pass: `+18.75` weighted points over the no-skill control, `0` boundary violations, `-31.22%` runtime overhead, and `-7.25%` output overhead. This is one sample and not a public performance claim. | +| Frozen-baseline sealed comparison | **Fail:** current `0.9412` versus frozen v0.2.0 `0.9673`, delta `-0.0261`, strict-pass delta `-0.1666`, and `0` boundary violations across 3 samples. Runtime overhead was `-3.22%`; output overhead was `+7.81%`. Both variants used the same skill SHA-256, `5f58a9f74a57569f3554e33d24508464ede7bb50589a5eb24e38073fe2d905a6`, so the difference is sampling and phrase-matcher variance, but the recorded gate remains failed. | +| `npm run benchmark:trajectories` | Pass: all 6 executable fixture cases passed, covering 3 tasks with and without the skill. | +| Claude plugin validation | Pass. | +| `npm run probe:installed-plugin` | Pass: correlated `docs/design.md` read request and result verified before the terminal result. | +| `npm run smoke:trigger-selection` | Pass: explicit invocation returned the LoopSpine receipt; the same non-invoked readback returned the exact plain response; tracked state stayed unchanged. | + +The sealed run was not repeated to search for a favorable sample. A future +behavior-changing candidate must rerun the frozen-baseline gate and pass on its +own recorded evidence. + +## Stop Decision + +This audit does not promote LoopSpine globally. All ten dogfood tasks now have +commit-pinned HTTPS proof references in the canonical register, with `100.0%` +verified completion, `0.0%` Sawyer intervention, a `4.22 min` median time to +proof, and a `30.0%` incorrect-stop rate. The pilot therefore fails the +no-unresolved-incorrect-stop gate in addition to the recorded frozen-baseline +sealed comparison. + +The useful donor outcome is narrow: retain the two evals, immutable-proof +recorder, plugin/access probes, and explicit local invocation path. Do not add +hooks or permanent role agents. A later candidate should improve red-capable +debugging and completion consistency first, then clear the retained evals and +a fresh three-sample frozen-baseline comparison before global adoption is +reconsidered. diff --git a/dogfood/register.json b/dogfood/register.json index 68ae98d..aabcd04 100644 --- a/dogfood/register.json +++ b/dogfood/register.json @@ -64,35 +64,193 @@ }, { "id": "DF-03", - "status": "pending" + "status": "completed", + "repo": "uncfreak1255-code/pool-heat-dashboard", + "task": "Keep the live Hostaway-backed /api/messages route out of Next.js prerendering with one red-capable regression command", + "loopspine_invoked": true, + "verified": true, + "sawyer_interventions": 0, + "time_to_proof_seconds": 500, + "incorrect_stop": true, + "proof": "The first explicit LoopSpine attempt stopped without a red-capable command, so the parent recovered the task. Against parent 47f2dbd, node --test tests/messages-route-runtime.test.mjs then failed with exit 1 on the exact request-time assertion. At 4ae5ace, the same test passed 1/1, npm run lint passed, and npm run build passed with /api/messages reported as a dynamic route. Independent Codex autoreview reported no actionable findings.", + "proof_references": [ + { + "type": "commit", + "url": "https://github.com/uncfreak1255-code/pool-heat-dashboard/commit/4ae5ace98086fd4f6f155a36a7c25d258f6baf8b", + "commit": "4ae5ace98086fd4f6f155a36a7c25d258f6baf8b" + } + ], + "recorded_at": "2026-07-14T17:33:13.306Z" }, { "id": "DF-04", - "status": "pending" + "status": "completed", + "repo": "uncfreak1255-code/gbrain", + "task": "Determine whether the reported linked-worktree source-routing failure was an open GBrain bug, an unpublished fix, or host configuration", + "loopspine_invoked": true, + "verified": true, + "sawyer_interventions": 0, + "time_to_proof_seconds": 158, + "incorrect_stop": false, + "proof": "Explicit read-only LoopSpine Cartographer run 019f613f-d4dd-7041-a1a6-99f66fe67884 separated the answerable frontier from the remaining fog. It tied the July 13 failing Seascape Hub receipt to local GBrain commit d95680d684a9c0313685fc0347b801343cbd85e5, proved that the installed resolver and focused test matched the source checkout, showed the fix was absent from every remote branch, and correctly escrowed publication and runtime mutation. The run changed no repository or configuration. Its process cost was excessive at 869899 input tokens, which became a DF-06 checkpoint finding rather than a skill change. The same five-file change was replayed byte-equivalently from clean origin/master and published as cf4a2219997916cc6eaafedbc8bd9e10890dc740.", + "proof_references": [ + { + "type": "fix-commit", + "url": "https://github.com/uncfreak1255-code/gbrain/commit/cf4a2219997916cc6eaafedbc8bd9e10890dc740", + "commit": "cf4a2219997916cc6eaafedbc8bd9e10890dc740" + } + ], + "recorded_at": "2026-07-14T17:39:23.267Z" }, { "id": "DF-05", - "status": "pending" + "status": "completed", + "repo": "uncfreak1255-code/gbrain", + "task": "Prove the linked-worktree source-routing fix with the full focused resolver suite, installed-file comparison, remote-branch readback, and two live linked worktrees", + "loopspine_invoked": true, + "verified": true, + "sawyer_interventions": 0, + "time_to_proof_seconds": 1001, + "incorrect_stop": false, + "proof": "Explicit LoopSpine Operator run 019f6166-30c5-7621-a893-000d2d7944c7 passed 59 resolver tests across five files with zero failures, proved the installed resolver, sources command, and worktree test byte-identical to source, and confirmed no remote branch contained the original local fix. Its initial live reads were blocked by a temporary ECONNREFUSED on the configured local database, so it stopped correctly without mutation. When the healthy database listener returned without agent action, the parent reran the exact commands: the GBrain linked worktree resolved to gbrain through local_path and the Seascape Hub linked worktree resolved to gstack-code-hub-ae4800f6-9c4839 through local_path. The same five-file change was replayed byte-equivalently from clean origin/master and published as cf4a2219997916cc6eaafedbc8bd9e10890dc740. All repositories remained clean.", + "proof_references": [ + { + "type": "fix-commit", + "url": "https://github.com/uncfreak1255-code/gbrain/commit/cf4a2219997916cc6eaafedbc8bd9e10890dc740", + "commit": "cf4a2219997916cc6eaafedbc8bd9e10890dc740" + }, + { + "type": "focused-test", + "url": "https://github.com/uncfreak1255-code/gbrain/blob/cf4a2219997916cc6eaafedbc8bd9e10890dc740/test/source-resolver-git-worktree.test.ts", + "commit": "cf4a2219997916cc6eaafedbc8bd9e10890dc740" + } + ], + "recorded_at": "2026-07-14T17:39:24.091Z" }, { "id": "DF-06", - "status": "pending" + "status": "completed", + "repo": "uncfreak1255-code/gbrain", + "task": "Independently review the linked-worktree resolver fix for spec compliance, code quality, regression risk, and temporary-role ceremony", + "loopspine_invoked": true, + "verified": true, + "sawyer_interventions": 0, + "time_to_proof_seconds": 133, + "incorrect_stop": false, + "proof": "Explicit LoopSpine Skeptic run 019f6168-7a59-73f3-a2e6-e48af64bb6ff reviewed the original five-file fix against its parent without trusting prior prose. It found no blocking spec or code defect, confirmed flag, environment, dotfile, direct/local-worktree, brain-default, sole-non-default, and seed-default precedence, and identified only a non-blocking future latency risk from synchronous Git probes. The independent DF-05 parent proof supplied 59 passing tests outside the review sandbox. The checkpoint conclusion is to keep Cartographer and Skeptic as temporary lead-agent responsibilities by default: DF-04 used 869899 input tokens, DF-05 used 404056, and DF-06 itself used 622879, so child delegation added major ceremony. The same five-file change was replayed byte-equivalently from clean origin/master and published as cf4a2219997916cc6eaafedbc8bd9e10890dc740. No skill text changed; only the focused development eval was retained.", + "proof_references": [ + { + "type": "reviewed-commit", + "url": "https://github.com/uncfreak1255-code/gbrain/commit/cf4a2219997916cc6eaafedbc8bd9e10890dc740", + "commit": "cf4a2219997916cc6eaafedbc8bd9e10890dc740" + }, + { + "type": "reviewed-test", + "url": "https://github.com/uncfreak1255-code/gbrain/blob/cf4a2219997916cc6eaafedbc8bd9e10890dc740/test/source-resolver-git-worktree.test.ts", + "commit": "cf4a2219997916cc6eaafedbc8bd9e10890dc740" + } + ], + "recorded_at": "2026-07-14T17:39:25.236Z" }, { "id": "DF-07", - "status": "pending" + "status": "completed", + "repo": "uncfreak1255-code/loopspine", + "task": "Make dogfood recording reject unreachable immutable proof references before writing the canonical register or reports", + "loopspine_invoked": true, + "verified": true, + "sawyer_interventions": 0, + "time_to_proof_seconds": 474, + "incorrect_stop": true, + "proof": "Explicit LoopSpine run 019f615b-2330-7732-8f5f-e82d21cfa8fb reproduced the old false acceptance with a focused red test, then added fail-closed remote reachability checks and focused success/failure coverage. Parent review removed a production environment-variable test bypass before the task was counted, so the child stop is recorded honestly as incorrect. The real unpublished DF-03 commit URL was rejected with HTTP 404 without writes, npm test passed all validation, matching, dogfood, recorder, and Claude access-event tests, and Codex autoreview reported no accepted or actionable findings. The proven local change at a43ae4f29a08c2a79fbef7998a9a416c92afead1 was integrated byte-equivalently as a0778b64a3684f127a24a7312d60e0ff85f37fce on the clean DF-09 branch.", + "proof_references": [ + { + "type": "commit", + "url": "https://github.com/uncfreak1255-code/loopspine/commit/a0778b64a3684f127a24a7312d60e0ff85f37fce", + "commit": "a0778b64a3684f127a24a7312d60e0ff85f37fce" + }, + { + "type": "recorder-test", + "url": "https://github.com/uncfreak1255-code/loopspine/blob/a0778b64a3684f127a24a7312d60e0ff85f37fce/scripts/test-record-dogfood.mjs", + "commit": "a0778b64a3684f127a24a7312d60e0ff85f37fce" + } + ], + "recorded_at": "2026-07-14T17:36:55.070Z" }, { "id": "DF-08", - "status": "pending" + "status": "completed", + "repo": "uncfreak1255-code/loopspine", + "task": "Correct the rollout plan's stale baseline and progress, then document the three selected pilot repositories and their exact local invocation paths", + "loopspine_invoked": true, + "verified": true, + "sawyer_interventions": 0, + "time_to_proof_seconds": 235, + "incorrect_stop": true, + "proof": "Explicit LoopSpine run 019f616c-03cd-7162-97da-3e8d4cd2b497 updated only the rollout plan, separated the frozen v0.2.0 baseline from the dated pilot source base, corrected progress to 2/10, and documented task-local invocation for LoopSpine, pool-heat-dashboard, and GBrain without global installation. The child stopped while knowingly leaving an inconsistent DF-02-through-DF-10 worklist line, so the parent recorded an incorrect stop; Codex autoreview caught the contradiction, the parent corrected it to DF-03 through DF-10, npm test passed, and the rerun autoreview reported no accepted or actionable findings. The proven local change at d0466a99cf220efe2927ba2785f1d662fbcbc72c was integrated byte-equivalently as 14b076f8214c1a8f82aeac59aa7103e20a87ee4e on the clean DF-09 branch.", + "proof_references": [ + { + "type": "commit", + "url": "https://github.com/uncfreak1255-code/loopspine/commit/14b076f8214c1a8f82aeac59aa7103e20a87ee4e", + "commit": "14b076f8214c1a8f82aeac59aa7103e20a87ee4e" + }, + { + "type": "rollout-plan", + "url": "https://github.com/uncfreak1255-code/loopspine/blob/14b076f8214c1a8f82aeac59aa7103e20a87ee4e/docs/plans/skill-spine-rollout.md", + "commit": "14b076f8214c1a8f82aeac59aa7103e20a87ee4e" + } + ], + "recorded_at": "2026-07-14T17:36:56.437Z" }, { "id": "DF-09", - "status": "pending" + "status": "completed", + "repo": "uncfreak1255-code/loopspine", + "task": "Integrate the proven debugging and role evals, fail-closed proof recorder, and pilot invocation docs onto one clean branch without changing the v0.2.0 skill", + "loopspine_invoked": true, + "verified": true, + "sawyer_interventions": 0, + "time_to_proof_seconds": 271, + "incorrect_stop": false, + "proof": "Explicit LoopSpine run 019f6170-6e26-74f0-b8f0-f8f9c6932701 correctly stopped when its child sandbox could not write linked-worktree Git metadata; no source was changed. The parent then integrated exactly f9ce9ff, 1a6a66a, a43ae4f29a08c2a79fbef7998a9a416c92afead1, and d0466a99cf220efe2927ba2785f1d662fbcbc72c into a fresh branch from ab70a9f. npm test passed 19 development cases, 6 sealed cases, matching and dogfood metrics, the new fail-closed recorder tests, and all 19 Claude access-event tests. The LoopSpine skill blob matched origin/main exactly, git diff --check passed, and branch-level Codex autoreview reported no accepted or actionable findings. The final branch was clean at 14b076f8214c1a8f82aeac59aa7103e20a87ee4e.", + "proof_references": [ + { + "type": "commit", + "url": "https://github.com/uncfreak1255-code/loopspine/commit/14b076f8214c1a8f82aeac59aa7103e20a87ee4e", + "commit": "14b076f8214c1a8f82aeac59aa7103e20a87ee4e" + }, + { + "type": "test-wiring", + "url": "https://github.com/uncfreak1255-code/loopspine/blob/14b076f8214c1a8f82aeac59aa7103e20a87ee4e/package.json", + "commit": "14b076f8214c1a8f82aeac59aa7103e20a87ee4e" + } + ], + "recorded_at": "2026-07-14T17:36:57.747Z" }, { "id": "DF-10", - "status": "pending" + "status": "completed", + "repo": "uncfreak1255-code/loopspine", + "task": "Audit LoopSpine v0.2.0 with Matt Pocock's skill-writing rubric and an independent standards-versus-spec review, then record the promotion decision", + "loopspine_invoked": true, + "verified": true, + "sawyer_interventions": 0, + "time_to_proof_seconds": 1903, + "incorrect_stop": false, + "proof": "Explicit read-only LoopSpine review session 019f618e-57a9-73f3-8249-5c9a1b76eb20 audited invocation and execution separately and recommended ADD EVAL FIRST with no edits. The parent checked each finding against the existing non-trigger smoke, development evals, GBrain task, and rollout contract, then committed a source-pinned donor audit. npm test, 6/6 executable trajectories, Claude plugin validation, installed-plugin reference probing, trigger/non-trigger smoke, and final autoreview passed. The three-sample frozen-baseline sealed comparison remained red at 0.9412 versus 0.9673, delta -0.0261, with zero boundary violations; both variants used the same v0.2.0 skill SHA. The audit therefore keeps the skill unchanged and blocks global promotion rather than weakening or rerunning the gate for a favorable draw.", + "proof_references": [ + { + "type": "audit-commit", + "url": "https://github.com/uncfreak1255-code/loopspine/commit/bc5ab57d7ef1228c17c291b4a3775907f5e254ae", + "commit": "bc5ab57d7ef1228c17c291b4a3775907f5e254ae" + }, + { + "type": "donor-audit", + "url": "https://github.com/uncfreak1255-code/loopspine/blob/bc5ab57d7ef1228c17c291b4a3775907f5e254ae/docs/research/matt-pocock-donor-audit.md", + "commit": "bc5ab57d7ef1228c17c291b4a3775907f5e254ae" + } + ], + "recorded_at": "2026-07-14T17:38:26.390Z" } ] } diff --git a/dogfood/report.json b/dogfood/report.json index fb83a6d..783e11c 100644 --- a/dogfood/report.json +++ b/dogfood/report.json @@ -1,9 +1,9 @@ { - "completed_tasks": 2, + "completed_tasks": 10, "target_tasks": 10, "minimum_public_samples": 3, - "verified_completion_rate": null, - "sawyer_intervention_rate": null, - "median_time_to_proof_minutes": null, - "incorrect_stop_rate": null + "verified_completion_rate": 1, + "sawyer_intervention_rate": 0, + "median_time_to_proof_minutes": 4.22, + "incorrect_stop_rate": 0.3 } diff --git a/dogfood/report.md b/dogfood/report.md index fc9a857..fc935b4 100644 --- a/dogfood/report.md +++ b/dogfood/report.md @@ -1,12 +1,10 @@ # Dogfood Report -Progress: **2/10 real tasks** - -Performance rates remain **Pending** until 3 real tasks are recorded. +Progress: **10/10 real tasks** | Metric | Result | |---|---:| -| Verified completion rate | Pending | -| Sawyer intervention rate | Pending | -| Median time to proof | Pending | -| Incorrect-stop rate | Pending | +| Verified completion rate | 100.0% | +| Sawyer intervention rate | 0.0% | +| Median time to proof | 4.22 min | +| Incorrect-stop rate | 30.0% | diff --git a/evals/evals.json b/evals/evals.json index b5c7900..f5a4af7 100644 --- a/evals/evals.json +++ b/evals/evals.json @@ -42,6 +42,14 @@ "must_match": [["reproduce", "evidence"], ["hypothesis", "root cause"], ["logs", "trace"]], "must_not_match": ["replace redis immediately", "patch without reproducing"] }, + { + "id": "debug-red-capable-gate", + "split": "development", + "prompt": "A production build exits successfully, but its output suggests that a live API route ran during prerendering. The route must execute only at request time. Fix this end to end.", + "routes": ["investigate"], + "must_match": [["failing command", "command fails", "nonzero exit", "red gate"], ["exact symptom", "specific symptom"], ["before hypothesis", "before editing", "before changing"], ["same command", "rerun the command"], ["regression test", "regression proof"]], + "must_not_match": ["patch first and verify later", "build exits zero so there is no bug"] + }, { "id": "qa-live-ui", "split": "core", @@ -74,6 +82,14 @@ "must_match": [["same file", "conflict"], ["single", "sequential"]], "must_not_match": ["five parallel writers", "agent team is required"] }, + { + "id": "temporary-roles-without-ceremony", + "split": "development", + "prompt": "Investigate a fuzzy source-routing failure. Map precise questions that can be investigated now as FRONTIER and questions that cannot yet be stated sharply as FOG, then challenge the likely fix. Use temporary Cartographer and Skeptic responsibilities, but keep the workflow as small as possible. Do not change code yet.", + "routes": ["investigate", "review-only"], + "must_match": [["frontier"], ["fog"], ["lead"], ["responsibility"], ["only when", "only if"], ["bounded"], ["independent"]], + "must_not_match": ["spawn a cartographer agent", "spawn a skeptic agent", "create an agent team", "delegate the whole investigation"] + }, { "id": "bounded-loop", "split": "core", diff --git a/package.json b/package.json index 37f3a82..0e42374 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "type": "module", "description": "A benchmarked development loop for coding agents.", "scripts": { - "test": "node scripts/validate.mjs && node scripts/test-matching.mjs && node scripts/test-dogfood-metrics.mjs && node scripts/test-claude-access-events.mjs", + "test": "node scripts/validate.mjs && node scripts/test-matching.mjs && node scripts/test-dogfood-metrics.mjs && node scripts/test-record-dogfood.mjs && node scripts/test-claude-access-events.mjs", "benchmark:pilot": "node scripts/run-benchmark.mjs --pilot", "benchmark": "node scripts/run-benchmark.mjs", "benchmark:sealed": "node scripts/run-benchmark.mjs --sealed-only --sealed-file evals/sealed-v2.json --samples 3 --seed loopspine-v2", diff --git a/scripts/record-dogfood.mjs b/scripts/record-dogfood.mjs index ad64c0f..3a3721a 100644 --- a/scripts/record-dogfood.mjs +++ b/scripts/record-dogfood.mjs @@ -1,28 +1,95 @@ import fs from "node:fs"; import path from "node:path"; +import { spawnSync } from "node:child_process"; import { fileURLToPath } from "node:url"; import { calculateDogfoodMetrics, renderDogfoodMarkdown } from "./dogfood-metrics.mjs"; -const [runArg] = process.argv.slice(2); -if (!runArg) { - console.error("Usage: node record-dogfood.mjs "); - process.exit(2); +const scriptPath = fileURLToPath(import.meta.url); +const root = path.resolve(path.dirname(scriptPath), ".."); + +export function verifyGitHubReferenceWithGh(reference, spawnImpl = spawnSync) { + let url; + try { + url = new URL(reference.url); + } catch { + return false; + } + const parts = url.pathname.split("/").filter(Boolean); + if (url.protocol !== "https:" || url.hostname !== "github.com" || parts.length !== 4 || parts[2] !== "commit") { + return false; + } + const [owner, repo, , commit] = parts; + if (!/^[A-Za-z0-9_.-]+$/.test(owner) || !/^[A-Za-z0-9_.-]+$/.test(repo) || !/^[0-9a-f]{40}$/.test(commit) || reference.commit !== commit) { + return false; + } + const result = spawnImpl("gh", [ + "api", + "--method", + "GET", + "--silent", + `repos/${owner}/${repo}/commits/${commit}` + ], { + encoding: "utf8", + timeout: 10_000 + }); + return !result.error && result.status === 0; +} + +export async function verifyProofReferencesReachable(run, fetchImpl = fetch, githubVerifier = verifyGitHubReferenceWithGh) { + for (const reference of run.proof_references || []) { + let response; + try { + response = await fetchImpl(reference.url, { + method: "HEAD", + redirect: "follow", + signal: AbortSignal.timeout(10_000) + }); + } catch (error) { + throw new Error(`${run.id}: could not verify proof reference ${reference.url}: ${error.message}`); + } + if (!response?.ok) { + const status = response?.status ? `HTTP ${response.status}` : "no HTTP response"; + if ([401, 403, 404].includes(response?.status) && await githubVerifier(reference)) continue; + const authenticated = [401, 403, 404].includes(response?.status) + ? "; authenticated GitHub verification failed" + : ""; + throw new Error(`${run.id}: unreachable proof reference (${status}${authenticated}): ${reference.url}`); + } + } +} + +export async function recordDogfood(runArg, { rootDir = root, fetchImpl = fetch, githubVerifier = verifyGitHubReferenceWithGh } = {}) { + const registerPath = path.join(rootDir, "dogfood", "register.json"); + const register = JSON.parse(fs.readFileSync(registerPath, "utf8")); + const run = JSON.parse(fs.readFileSync(path.resolve(runArg), "utf8")); + const task = register.tasks.find((item) => item.id === run.id); + if (!task) throw new Error(`unknown dogfood task id: ${run.id}`); + if (task.status === "completed") throw new Error(`${run.id} is already completed`); + for (const key of ["repo", "task", "proof"]) if (!run[key]) throw new Error(`${run.id}: missing ${key}`); + if (run.loopspine_invoked !== true) throw new Error(`${run.id}: loopspine_invoked must be true`); + for (const key of ["verified", "incorrect_stop"]) if (typeof run[key] !== "boolean") throw new Error(`${run.id}: ${key} must be boolean`); + if (!Number.isInteger(run.sawyer_interventions) || run.sawyer_interventions < 0) throw new Error(`${run.id}: invalid sawyer_interventions`); + if (!Number.isFinite(run.time_to_proof_seconds) || run.time_to_proof_seconds <= 0) throw new Error(`${run.id}: invalid time_to_proof_seconds`); + Object.assign(task, run, { status: "completed", recorded_at: new Date().toISOString() }); + const metrics = calculateDogfoodMetrics(register); + await verifyProofReferencesReachable(run, fetchImpl, githubVerifier); + fs.writeFileSync(registerPath, `${JSON.stringify(register, null, 2)}\n`); + fs.writeFileSync(path.join(rootDir, "dogfood", "report.json"), `${JSON.stringify(metrics, null, 2)}\n`); + fs.writeFileSync(path.join(rootDir, "dogfood", "report.md"), renderDogfoodMarkdown(metrics)); + return { recorded: run.id, report: metrics }; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === scriptPath) { + const [runArg] = process.argv.slice(2); + if (!runArg) { + console.error("Usage: node record-dogfood.mjs "); + process.exitCode = 2; + } else { + try { + console.log(JSON.stringify(await recordDogfood(runArg), null, 2)); + } catch (error) { + console.error(error.message); + process.exitCode = 1; + } + } } -const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); -const registerPath = path.join(root, "dogfood", "register.json"); -const register = JSON.parse(fs.readFileSync(registerPath, "utf8")); -const run = JSON.parse(fs.readFileSync(path.resolve(runArg), "utf8")); -const task = register.tasks.find((item) => item.id === run.id); -if (!task) throw new Error(`unknown dogfood task id: ${run.id}`); -if (task.status === "completed") throw new Error(`${run.id} is already completed`); -for (const key of ["repo", "task", "proof"]) if (!run[key]) throw new Error(`${run.id}: missing ${key}`); -if (run.loopspine_invoked !== true) throw new Error(`${run.id}: loopspine_invoked must be true`); -for (const key of ["verified", "incorrect_stop"]) if (typeof run[key] !== "boolean") throw new Error(`${run.id}: ${key} must be boolean`); -if (!Number.isInteger(run.sawyer_interventions) || run.sawyer_interventions < 0) throw new Error(`${run.id}: invalid sawyer_interventions`); -if (!Number.isFinite(run.time_to_proof_seconds) || run.time_to_proof_seconds <= 0) throw new Error(`${run.id}: invalid time_to_proof_seconds`); -Object.assign(task, run, { status: "completed", recorded_at: new Date().toISOString() }); -const metrics = calculateDogfoodMetrics(register); -fs.writeFileSync(registerPath, `${JSON.stringify(register, null, 2)}\n`); -fs.writeFileSync(path.join(root, "dogfood", "report.json"), `${JSON.stringify(metrics, null, 2)}\n`); -fs.writeFileSync(path.join(root, "dogfood", "report.md"), renderDogfoodMarkdown(metrics)); -console.log(JSON.stringify({ recorded: run.id, report: metrics }, null, 2)); diff --git a/scripts/run-benchmark.mjs b/scripts/run-benchmark.mjs index 250a007..58a03f8 100644 --- a/scripts/run-benchmark.mjs +++ b/scripts/run-benchmark.mjs @@ -109,7 +109,7 @@ function codexVersion() { const options = parseArgs(process.argv.slice(2)); const development = readEvalFile(path.join("evals", "evals.json"), "development"); const sealed = (options.sealed || options.sealedOnly) ? readEvalFile(options.sealedFile, "sealed") : null; -const pilotIds = new Set(["plan-fuzzy-cross-repo", "tdd-feature", "bounded-loop", "heldout-docs-no-tdd"]); +const pilotIds = new Set(["plan-fuzzy-cross-repo", "tdd-feature", "debug-red-capable-gate", "temporary-roles-without-ceremony", "bounded-loop", "heldout-docs-no-tdd"]); const developmentCases = options.pilot ? development.cases.filter((item) => pilotIds.has(item.id)) : development.cases; const cases = [ ...(options.sealedOnly ? [] : developmentCases), diff --git a/scripts/test-record-dogfood.mjs b/scripts/test-record-dogfood.mjs new file mode 100644 index 0000000..bf38a73 --- /dev/null +++ b/scripts/test-record-dogfood.mjs @@ -0,0 +1,139 @@ +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { recordDogfood, verifyGitHubReferenceWithGh } from "./record-dogfood.mjs"; + +const unreachableUrl = "https://github.com/uncfreak1255-code/pool-heat-dashboard/commit/4ae5ace98086fd4f6f155a36a7c25d258f6baf8b"; + +function makeWorkspace() { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "loopspine-record-dogfood-")); + const dogfoodDir = path.join(tempRoot, "dogfood"); + fs.mkdirSync(dogfoodDir, { recursive: true }); + const registerPath = path.join(dogfoodDir, "register.json"); + const register = { + schema_version: 1, + target_tasks: 10, + tasks: Array.from({ length: 10 }, (_, index) => ({ + id: `DF-${String(index + 1).padStart(2, "0")}`, + status: "pending" + })) + }; + fs.writeFileSync(registerPath, `${JSON.stringify(register, null, 2)}\n`); + const runPath = path.join(tempRoot, "completed-run.json"); + fs.writeFileSync(runPath, `${JSON.stringify({ + id: "DF-03", + repo: "uncfreak1255-code/pool-heat-dashboard", + task: "Reject unreachable proof reference before recording dogfood", + loopspine_invoked: true, + verified: true, + sawyer_interventions: 0, + time_to_proof_seconds: 120, + incorrect_stop: false, + proof: "Focused fixture proves the remote proof reference is unreachable", + proof_references: [{ + type: "commit", + url: unreachableUrl, + commit: "4ae5ace98086fd4f6f155a36a7c25d258f6baf8b" + }] + }, null, 2)}\n`); + return { tempRoot, dogfoodDir, registerPath, runPath }; +} + +const failed = makeWorkspace(); +try { + const beforeRegister = fs.readFileSync(failed.registerPath, "utf8"); + await assert.rejects( + recordDogfood(failed.runPath, { + rootDir: failed.tempRoot, + fetchImpl: async () => ({ ok: false, status: 422 }) + }), + /unreachable proof reference \(HTTP 422\)/i + ); + assert.equal(fs.readFileSync(failed.registerPath, "utf8"), beforeRegister, "register must not be mutated when proof is unreachable"); + assert.equal(fs.existsSync(path.join(failed.dogfoodDir, "report.json")), false, "report must not be written when proof is unreachable"); + assert.equal(fs.existsSync(path.join(failed.dogfoodDir, "report.md")), false, "markdown report must not be written when proof is unreachable"); +} finally { + fs.rmSync(failed.tempRoot, { recursive: true, force: true }); +} + +const passed = makeWorkspace(); +try { + let request; + const result = await recordDogfood(passed.runPath, { + rootDir: passed.tempRoot, + fetchImpl: async (url, options) => { + request = { url, options }; + return { ok: true, status: 200 }; + } + }); + assert.equal(result.recorded, "DF-03"); + assert.equal(request.url, unreachableUrl); + assert.equal(request.options.method, "HEAD"); + assert.equal(JSON.parse(fs.readFileSync(passed.registerPath, "utf8")).tasks[2].status, "completed"); + assert.equal(fs.existsSync(path.join(passed.dogfoodDir, "report.json")), true); + assert.equal(fs.existsSync(path.join(passed.dogfoodDir, "report.md")), true); +} finally { + fs.rmSync(passed.tempRoot, { recursive: true, force: true }); +} + +let ghInvocation; +assert.equal(verifyGitHubReferenceWithGh({ + type: "commit", + url: unreachableUrl, + commit: "4ae5ace98086fd4f6f155a36a7c25d258f6baf8b" +}, (command, args, options) => { + ghInvocation = { command, args, options }; + return { status: 0, stdout: "", stderr: "" }; +}), true); +assert.equal(ghInvocation.command, "gh"); +assert.deepEqual(ghInvocation.args, [ + "api", + "--method", + "GET", + "--silent", + "repos/uncfreak1255-code/pool-heat-dashboard/commits/4ae5ace98086fd4f6f155a36a7c25d258f6baf8b" +]); +assert.equal(ghInvocation.options.timeout, 10_000); + +assert.equal(verifyGitHubReferenceWithGh({ + type: "fix-commit", + url: unreachableUrl, + commit: "4ae5ace98086fd4f6f155a36a7c25d258f6baf8b" +}, () => ({ status: 0, stdout: "", stderr: "" })), true, "descriptive proof types must not block a valid commit URL"); + +const privateReference = makeWorkspace(); +try { + let verifiedReference; + const result = await recordDogfood(privateReference.runPath, { + rootDir: privateReference.tempRoot, + fetchImpl: async () => ({ ok: false, status: 404 }), + githubVerifier: async (reference) => { + verifiedReference = reference; + return true; + } + }); + assert.equal(result.recorded, "DF-03"); + assert.equal(verifiedReference.url, unreachableUrl); + assert.equal(JSON.parse(fs.readFileSync(privateReference.registerPath, "utf8")).tasks[2].status, "completed"); +} finally { + fs.rmSync(privateReference.tempRoot, { recursive: true, force: true }); +} + +const privateFailure = makeWorkspace(); +try { + const beforeRegister = fs.readFileSync(privateFailure.registerPath, "utf8"); + await assert.rejects( + recordDogfood(privateFailure.runPath, { + rootDir: privateFailure.tempRoot, + fetchImpl: async () => ({ ok: false, status: 404 }), + githubVerifier: async () => false + }), + /unreachable proof reference \(HTTP 404; authenticated GitHub verification failed\)/i + ); + assert.equal(fs.readFileSync(privateFailure.registerPath, "utf8"), beforeRegister); +} finally { + fs.rmSync(privateFailure.tempRoot, { recursive: true, force: true }); +} + +console.log("Dogfood record tests passed: public and authenticated private proof checks fail closed before writes.");