Skip to content

docs: sync coverage figures to the generated report (351/443) #311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
doublegate merged 2 commits into from
Aug 1, 2026
+34 −12
Merged

docs: sync coverage figures to the generated report (351/443) #311

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
69be752
docs: sync coverage figures to the generated report (351/443)
doublegate Aug 1, 2026
cb274f4
docs: the scene tally in plan.md §5 had drifted too (53 -> 54)
doublegate Aug 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed

- **CORRECTION: the battery covers 351 of 443 assertions, not 350.** The `E5.06` entry below says
coverage was "350 either way"; the regenerated `docs/accuracysnes-coverage.md` reads **297 on-cart
+ 54 scene-only = 351**. `docs/STATUS.md` and `docs/accuracysnes-plan.md` are synced to the
generated figure, along with three other hand-maintained lines that had drifted behind this
cycle's work: the reference list (now **Mesen2, snes9x and ares**, with MesenCE named separately as
the compositor blueprint rather than as a cross-validation reference), Mesen2's known-failure
rationale (`F1.03` alone, and *why* — it clocks both ports out of one latch), and the scene tally
(54, on both scene hosts). The generated report is the authority precisely because these drift;
this is the periodic reconciliation, not a coverage change.

### Added

- **`E3.06` rewritten to poll and accumulate — and ares agrees with it exactly.** The row read
Expand Down
13 changes: 10 additions & 3 deletions docs/STATUS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ compositor (`docs/adr/0014`, T-CA-10) becomes the sole renderer — its compose/
per-dot against live registers (mid-line CGRAM/OAM/`INIDISP` at dot resolution; BG-fetch-ahead is
Phase 4c, not yet landed) — and the first-party AccuracySNES
cartridge matures into a usable instrument (AccuracyCoin-style paged menu + automatic skyline
results + per-test B-skip + a Select WRAM debug viewer) with its battery at **350 of 443** dossier
assertions (297 on-cart + 53 rendered scenes).
results + per-test B-skip + a Select WRAM debug viewer) with its battery at **351 of 443** dossier
assertions (297 on-cart + 54 rendered scenes).
Comment on lines +26 to +27

@coderabbitai coderabbitai Bot Aug 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use dossier coverage terminology.

Line 26-27 calls 351/443 the battery size. The plan distinguishes 338 battery tests from 351 dossier assertions. State that the dossier coverage is 351/443 to prevent readers from treating 351 as the test count.

Proposed wording 
-results + per-test B-skip + a Select WRAM debug viewer) with its battery at **351 of 443** dossier
-assertions (297 on-cart + 54 rendered scenes).
+results + per-test B-skip + a Select WRAM debug viewer) with dossier coverage at **351 of 443**
+assertions (297 on-cart + 54 rendered scenes).

Based on learnings: battery test count and dossier assertion coverage are intentionally non-equivalent.

As per path instructions: Docs are the spec, not a history log.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
results + per-test B-skip + a Select WRAM debug viewer) with its battery at **351 of 443** dossier
assertions (297 on-cart + 54 rendered scenes).
results + per-test B-skip + a Select WRAM debug viewer) with dossier coverage at **351 of 443**
assertions (297 on-cart + 54 rendered scenes).
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/STATUS.md` around lines 26 - 27, Update the status wording around the
“351 of 443” figure to identify it as dossier assertion coverage, not the
battery test count; preserve the distinction between the 338 battery tests and
351 dossier assertions.

Sources: Path instructions, Learnings

**`v1.22.0 "Horizon"` fixes the long-standing DSP-1 continuous-mode Mode-7 flat floor** (Pilotwings
flight, SMK track): the shared NEC-DSP host-sync (`run_until_rqm`) stopped one host-write too early,
so reading a host input word raised RQM before the firmware cleared DRC to 16-bit — mis-framing the
Expand Down Expand Up @@ -216,6 +216,13 @@ pass rate, with `Contested`/`Novel` recorded but never counted. Behaviour hardwa
define is captured as a **golden vector**, never scored. **Real-hardware validation has not been
done and is the honest ceiling on its authority.**

As of `v1.29.0` a **headless ares host** (`scripts/accuracysnes/ares_host/`) runs the battery as a
third reference, which matters because this project's provenance rule counts ares and bsnes as one:
several rows sat at 2-vs-1 with no way to break the tie. It also produced the first defect
AccuracySNES has found in a *reference* emulator rather than in this one — ares' `$F1` handler
negates the timer-2 counter reset, so any later `$F1` write zeroes `T2OUT`, while its own timers 0
and 1 use the un-negated form.

Comment on lines +219 to +225

@coderabbitai coderabbitai Bot Aug 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Align all ares claims with the executable validation path.

The supplied scripts/accuracysnes/crossval.sh explicitly excludes ares and states that no headless or memory-dump path exists. The changed documentation and changelog therefore report results that the shown validation path cannot produce. Update the host, driver, and citations together. If the ares claims remain, use “three reference emulators” unless the provenance rule supports “independent.”

  • docs/STATUS.md#L219-L225: add the ares host to the executable validation path, or remove the headless-host and defect claims.
  • docs/STATUS.md#L244-L244: remove ares from the cross-validation list until the runner produces the result, and replace “independent references” unless justified by the provenance model.
  • docs/accuracysnes-plan.md#L20-L24: do not cite scripts/accuracysnes/crossval.sh for ares divergences until the script runs ares.
  • docs/accuracysnes-plan.md#L746-L751: report ares: OK (4 known) and all-host scene results only after the repository contains that runnable validation path.
  • CHANGELOG.md#L18-L21: document ares as a reference only when the corresponding host and driver are shipped.

As per path instructions: Docs are the spec, not a history log.

📍 Affects 3 files
  • docs/STATUS.md#L219-L225 (this comment)
  • docs/STATUS.md#L244-L244
  • docs/accuracysnes-plan.md#L20-L24
  • docs/accuracysnes-plan.md#L746-L751
  • CHANGELOG.md#L18-L21
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/STATUS.md` around lines 219 - 225, Align the documentation with the
currently executable validation path by removing unsupported ares claims unless
a runnable host and driver are added. In docs/STATUS.md lines 219-225 remove the
headless-host and defect claims; in docs/STATUS.md line 244 remove ares from the
cross-validation list and replace “independent references” with
provenance-accurate wording; in docs/accuracysnes-plan.md lines 20-24 and
746-751 remove ares-specific and all-host results; and in CHANGELOG.md lines
18-21 document ares only once its host and driver are shipped.

Source: Path instructions

The composed multi-suite battery below remains in force alongside it — AccuracySNES adds a layer,
it does not replace the per-opcode oracles. The accuracy story here is instead a **composed multi-layer battery** across
independently-sourced suites (`docs/testing-strategy.md`). Rather than force these heterogeneous
Expand All @@ -234,7 +241,7 @@ tracked here, always current, reaffirmed every release:
| Core/Curated coprocessors (oracle-gated) | ✅ **3 / 3, honesty gate green** | DSP-1 (4 commercial ROMs), Super FX/GSU (58 Krom ROMs + per-opcode suite), SA-1 (18 commercial carts) — `ORACLE_COPROCESSORS` |
| BestEffort coprocessors, real-title validated | ✅ **10 / 11** | DSP-2, DSP-4, ST010, S-DD1, CX4, OBC1 boot a real commercial title to real gameplay content. **DSP-3 (SD Gundam GX) + ST011 (2-dan Morita Shougi)** are liveness/determinism-validated — detection + `host_accesses > 0` + bit-identical framebuffer (`dsp3_st011_oncart`). **ST018 (Nidan Morita Shogi 2) + S-RTC (Daikaijuu Monogatari II)** are detection + boot + determinism-validated against their real carts (`srtc_st018_oncart`), but their coprocessor's core function is **usage-gated** (the ST018 shogi AI runs only on the computer's move; the S-RTC clock is read at specific moments), so it is not exercised in a headless boot |
| BestEffort coprocessors, no booting dump | ⚠️ **1 / 11** | SPC7110 — **two** local Tengai Makyou Zero dumps both fail to boot to content: the 7 MiB one is a fan-translation ROM hack needing a patch-only memory region no cartridge has (`docs/audit/spc7110-boot-crash-2026-07-08.md`); a 5 MiB dump (sha256 `8620203d…`) freezes at a near-blank screen with zero coprocessor activity, and does not match the documented-good original (`69d06a3f…`). The correct original-cartridge dump remains the ROM-sourcing gap (`docs/rom-test-corpus.md`) |
| AccuracySNES (first-party battery) | ✅ **350 / 443 assertions covered, 100% on-cart pass** | A 338-test self-scoring battery spanning Groups A-G. **350 of 443** dossier assertions covered: **297 on-cart** + **53 via rendered scenes** (`docs/adr/0013`), kept as separate columns on purpose (`docs/accuracysnes-coverage.md`, regenerated with the ROM so it cannot drift). Every scored row is inject-verified for non-vacuity and cross-validated headlessly against **Mesen2, snes9x, and MesenCE** (the last built this cycle as the per-dot compositor's blueprint and exact-frame oracle). Provenance gate green. Ships an AccuracyCoin-style on-cart UI as of `v1.21.0` (paged menu + automatic skyline results + per-test B-skip + a Select WRAM debug viewer). Remaining ~93 assertions are compositor-gated hi-res, second-image G-rows, treacherous CPU/dot-model timing, or provably uncoverable — though "uncoverable" has twice proved too strong (`E5.06` and `B2.02`/`B2.03` all landed after being parked) (`docs/accuracysnes-coverability-audit-2026-07-23.md`) |
| AccuracySNES (first-party battery) | ✅ **351 / 443 assertions covered, 100% on-cart pass** | A 338-test self-scoring battery spanning Groups A-G. **351 of 443** dossier assertions covered: **297 on-cart** + **54 via rendered scenes** (`docs/adr/0013`), kept as separate columns on purpose (`docs/accuracysnes-coverage.md`, regenerated with the ROM so it cannot drift). Every scored row is inject-verified for non-vacuity and cross-validated headlessly against **Mesen2, snes9x and ares** — three independent references as of `v1.29.0`, with MesenCE additionally serving as the per-dot compositor's blueprint and exact-frame oracle. Provenance gate green. Ships an AccuracyCoin-style on-cart UI as of `v1.21.0` (paged menu + automatic skyline results + per-test B-skip + a Select WRAM debug viewer). Remaining ~92 assertions are compositor-gated hi-res, second-image G-rows, treacherous CPU/dot-model timing, or provably uncoverable — though "uncoverable" has twice proved too strong (`E5.06` and `B2.02`/`B2.03` all landed after being parked) (`docs/accuracysnes-coverability-audit-2026-07-23.md`) |
| Determinism contract | ✅ **proven** | bit-identical framebuffer/audio across runs; save-state round-trip proven across all three board tiers (no-coprocessor, Curated, BestEffort) |

**Named residuals, tracked not hidden:** the 65816 `e1.e` divergence (`docs/adr/0002`);
Expand Down
21 changes: 12 additions & 9 deletions docs/accuracysnes-plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,14 @@ AccuracySNES closed ticket **T-04**. The follow-on tickets minted here are **T-0
| | |
|---|---|
| Tests | **338** (scoring + golden vectors + region SKIP per image) — *tests, not assertions; see the note below the table* |
| Assertion coverage | **350 of 443** dossier assertions — **297 on-cart** + **53 rendered scenes**, kept as separate columns (`docs/accuracysnes-coverage.md`) |
| Rendered scenes | **54** declared and all cross-validated (`docs/adr/0013`); **53** are the dossier rows they are the only cover for |
| Assertion coverage | **351 of 443** dossier assertions — **297 on-cart** + **54 rendered scenes**, kept as separate columns (`docs/accuracysnes-coverage.md`) |
| Rendered scenes | **54** declared, all blessed and matching on both scene hosts (`docs/adr/0013`); **54** dossier rows have a scene as their only cover |
| Pass rate | **100.00%** on-cart, floor enforced at 1.00 by `tests/accuracysnes.rs` |
| Cross-validated | RustySNES and Mesen2 agree on every test but two `PAD2_CONTRACT` rows the Lua runner cannot drive; snes9x agrees on every test but a handful of recorded reference bugs with citations in `scripts/accuracysnes/crossval.sh`; a headless **MesenCE** (this cycle) is the per-dot compositor's blueprint + exact-frame oracle. All images. |
| Cross-validated | **Three references** as of `v1.29.0`. Mesen2 agrees on every test but `F1.03`, which clocks both ports out of one latch and so needs the port-2 input its Lua runner cannot drive; snes9x has 14 recorded divergences and **ares** 4, each with a citation in `scripts/accuracysnes/crossval.sh`. A headless **MesenCE** is separately the per-dot compositor's blueprint + exact-frame oracle. All images. |
| Groups shipped | **A** (65C816) · **B** (5A22) · **C** (PPU, on-cart and rendered) · **D** (DMA/HDMA) · **E** (SPC700 + S-DSP) · **F** (controller ports) · **G** (cartridge/memory map) — all seven, all partial |
| On-cart UI | AccuracyCoin-style paged menu + automatic skyline results + per-test B-skip + a Select WRAM debug viewer (`v1.21.0`) |
| Defects found in this emulator | **12+** — see §5 |
| Defects found in a *reference* emulator | **1** — ares' `$F1` handler negates the timer-2 counter reset (`scripts/accuracysnes/ares_host/README.md`) |

These counts are maintained by hand and will drift. **`docs/accuracysnes-coverage.md` is the
authority**: it is regenerated with the ROM, so it cannot.
Expand Down Expand Up @@ -742,10 +743,12 @@ Chosen 2026-08-01, now that both halves of the Mesen2 oracle arbitrate. Two corr
| `E9.02` | noise output is **highpass-filtered** `[ERRATA]` | DSP-observable via OUTX on a NON voice; errata-flagged |
| `E5.06` | BRR 15-bit wrap: clamp to 16 bits, then `+4000h..+7FFFh → -4000h..-1` | reuses the landed `E5` decoder scaffolding |

**All three landed 2026-08-01.** Battery 338 tests, 100% on-cart, and both references agree:
`snes9x: OK (14 known divergence(s))`, `Mesen2: OK (2 known divergence(s))`, 53/53 scenes on each.
`MESEN2_KNOWN_FAILURES` came **down** from 3 to 2 in the process, because the redesign below removed
the one Mesen2 divergence that was ours rather than the harness's.
**All three landed 2026-08-01.** Battery 338 tests, 100% on-cart, and the references agree:
`snes9x: OK (14 known)`, `Mesen2: OK (1 known)`, `ares: OK (4 known)`, 54/54 scenes on each scene
host. `MESEN2_KNOWN_FAILURES` came **down** from 3, first to 2 because the redesign below removed the
one Mesen2 divergence that was ours rather than the harness's, and then to 1 when `F1.10`'s Mesen2
verdict proved phase-fragile (it flipped to passing when an unrelated `E3.06` rewrite moved the
cart's execution phase) and was retracted rather than re-recorded.

### `E8.01` — two rejected drafts, and what they were really measuring

Expand Down Expand Up @@ -2337,8 +2340,8 @@ exists.
stay in their own tier. `crossval.sh` gates on them, and per rule 4 a golden is committed only
once the references agree.

**Status: 53 scenes blessed** (the regenerated `docs/accuracysnes-coverage.md` is authoritative — this
hand-maintained figure had drifted from an earlier "41"), covering the scene tier across `C4`-`C8`,
**Status: 54 scenes blessed** (the regenerated `docs/accuracysnes-coverage.md` is authoritative — this
hand-maintained figure had drifted from an earlier "41", and again from "53"), covering the scene tier across `C4`-`C8`,
`C10`, `C11` and `C12`. The
first three disagreed with the references on first run, and in all three cases RustySNES was
wrong: the BG vertical fetch was a line late, and mosaic quantised the BG row instead of the
Expand Down
Loading