From 6561a0cc40274847b35d3e8c02fb7ee471fdddc2 Mon Sep 17 00:00:00 2001 From: DoubleGate Date: Sat, 1 Aug 2026 11:03:36 -0400 Subject: [PATCH] =?UTF-8?q?feat(accuracysnes):=20ares=20as=20a=20third=20r?= =?UTF-8?q?eference=20=E2=80=94=20and=20retract=20the=20A2.10=20finding?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RETRACTION, and it is the important half. The plan recorded "A2.10 -- Mesen2 is the outlier, 2-vs-1" as the battery's only unexplained reference disagreement, and PR #304 said ares settled it 3-vs-1. Measured: RustySNES pass Mesen2 PASS (its failing set is F1.03 and F1.10, nothing else, identical on six runs) ares pass snes9x FAIL code 1 -- and documented, the FIRST entry in SNES9X_KNOWN_FAILURES So the outlier is snes9x, the failure is already explained, and there was never a disagreement on this row. The old text also asserted "A2.10 is not among snes9x's 14 known divergences"; it is, at crossval.sh:80. The original reading took a catalogue index off one host's output and attributed it to another, and nothing checked it because no per-row measurement existed. That is the second published claim in one day undone by keying on catalogue indices instead of row names. An index is only meaningful next to the host that produced it and the catalogue that was current, and neither travels with the number. The ares host was therefore built to settle a row that did not need settling. It still earned itself: it found a real bug in ares' SMP. ares is now wired in as a third reference. cross-validation reports "3 reference(s) agree with the cart". ARES_KNOWN_FAILURES=5 carries per-row rationale like the other two constants -- C7.05/C7.10/F1.10 are rows snes9x already fails, E3.06/E8.02 cite the ares $F1 timer-2 reset bug. The block is opt-in and skips cleanly when the binary is absent, verified in both states. One bug caught on the way: counting failures with ("0x" $3) % 2 == 0 reported 337 failures, because POSIX awk does not parse "0x01" and the expression is 0 % 2 for every byte. A gate reporting catastrophe out of a parsing bug is a specific kind of dangerous; the fix matches the last hex digit and says why. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 28 ++++++++-- docs/accuracysnes-plan.md | 66 ++++++++++++------------ scripts/accuracysnes/ares_host/README.md | 10 ++-- scripts/accuracysnes/crossval.sh | 63 ++++++++++++++++++++++ 4 files changed, 129 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a538786a..12d88264 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **ares is wired into `crossval.sh` as a third reference.** `cross-validation: 3 reference(s) agree + with the cart`, with `ARES_KNOWN_FAILURES=5` carrying the same per-row rationale the other two + constants do: `C7.05`/`C7.10`/`F1.10` are rows snes9x already fails (ares corroborating it, not + standing alone), and `E3.06`/`E8.02` cite the ares `$F1` timer-2 reset bug. + + The block is **opt-in and skips cleanly**: the host is a C++ link against ares' static libraries + and takes minutes, so `crossval.sh` looks for a pre-built binary at `$ARES_HOST` and prints + `skip ares: build it with 'bash scripts/accuracysnes/ares_host/build.sh'` when it is absent — + the same shape as the snes9x and Mesen2 blocks, and verified in both states. + + One bug caught on the way, worth recording because of how it presented: counting the failing rows + with `("0x" $3) % 2 == 0` reported **337 failures**. POSIX awk does not parse `"0x01"`, so the + expression is `0 % 2` for every byte and every non-skipped row counted as a failure. A gate + reporting catastrophe out of a parsing bug is a specific kind of dangerous, so the fix matches on + the last hex digit and says why in a comment. + - **AccuracySNES found a bug in ares — the first time it has found one in a reference emulator.** `E3.06` and `E8.02` were the two divergences the third opinion could not attribute to a known snes9x failure, and both read **0** from every timer-2 slot they record. One mechanism explains @@ -63,9 +79,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 magic ACSN / done a5 / count 338 / passed 299 / failed 5 / skipped 1 / golden 33 ``` - Reproducible across runs. **ares passes `A2.10`** ("PEI does not page-wrap", catalogue index 11, - status `$01`), so with RustySNES and snes9x it is **3 against 1 with Mesen2 the outlier**, and the - row comes off the "unexplained, needs a fourth opinion" list it has been on since the oracle fix. + Reproducible across runs. ares passes `A2.10` (catalogue index 11, status `$01`). + + **RETRACTION.** This entry first said that made it "3 against 1 with Mesen2 the outlier". It does + not, because **Mesen2 passes `A2.10` too** — measured with `mesen_failing_set_probe.lua`, its + failing set is `F1.03` and `F1.10` and nothing else, identical on six runs. The host that fails + `A2.10` is **snes9x**, where it is documented as the *first* entry in `SNES9X_KNOWN_FAILURES`. So + there was never a disagreement on this row: the original reading took an index off one host's + output and attributed it to another. The ares host was built to settle a row that did not need + settling — and still earned itself, by finding a real ares bug that needed a third opinion to see. **Five rows where ares disagrees with the cart** — and **three of them are rows snes9x already fails**, which the tally alone does not show. `C7.10` and `F1.10` become **2-vs-2** (RustySNES + diff --git a/docs/accuracysnes-plan.md b/docs/accuracysnes-plan.md index dc3085b5..dbc526fc 100644 --- a/docs/accuracysnes-plan.md +++ b/docs/accuracysnes-plan.md @@ -839,38 +839,40 @@ cannot increment between two reads six cycles apart. Unless a distinguishing obs first, this is the shape [[accuracysnes-vacuous-test-trap]] exists to prevent — design the observable before writing the row. -**Before authoring any of them,** settle `A2.10` above. It is the only unexplained reference -disagreement in the battery, and a row batch authored while one row is unexplained risks attributing -a new failure to the wrong cause. - -#### First arbitrated finding: `A2.10` — Mesen2 is the outlier, and ares makes it 3-vs-1 - -The moment the oracle started working it produced a result the project has never been able to see. -`crossval.sh` reports `Mesen2: 1 failing test(s)`, catalogue index **11 = cart `A2.10`, "PEI does not -page-wrap"** (scored; source: WDC datasheet + superfamicom.org addressing notes; verdict slot -`$7EF02B`). - -RustySNES **passes** it and snes9x passes it too — `A2.10` is not among snes9x's 14 known -divergences. So this is **2-vs-1 with Mesen2 as the outlier**, which inverts the usual shape: the -standing heuristic warns that *RustySNES failing alone* means a real bug, and that is not what this -is. - -**SETTLED 2026-08-01 — ares passes it, so this is 3-vs-1.** The third opinion the paragraph below -asked for now exists: `scripts/accuracysnes/ares_host/` runs the battery under ares -(`magic ACSN`, `done a5`, 338 tests, reproducible across runs) and reports catalogue index 11 as -status `$01` — **pass**. With RustySNES and snes9x that is three implementations against Mesen2, and -the caveat that held this open — a harness bug upstream of an implementation produces the same -signature, as the `$F8`/`$F9` retraction proved — is answered: the fourth host is independent of all -three and agrees. - -`A2.10` can therefore become a `MESEN2_KNOWN_FAILURES` entry whenever the count is next revised. It -is not being added in the same change that established it, because `MESEN2_KNOWN_FAILURES` currently -reads 2 and matches; changing the constant and the reasoning at once would leave nothing to check -the change against. - -*The original reasoning, kept because it is why the row waited:* recording it as a known-good Mesen2 -divergence without a third opinion would have been exactly the mistake the `$F8`/`$F9` retraction -already cost this project once. +**~~Before authoring any of them, settle `A2.10` above.~~** Retracted with the section below: there +was never an `A2.10` disagreement to settle. The battery has **no** unexplained reference +disagreement — every failing row on every host is now measured and attributed. + +#### RETRACTED: the `A2.10` "arbitrated finding" was a host mix-up + +This section used to read *"`A2.10` — Mesen2 is the outlier, 2-vs-1"*, and called it the only +unexplained reference disagreement in the battery. **It was wrong in both directions**, and measuring +it took a probe that did not exist until 2026-08-01. + +| | `A2.10` (catalogue index 11) | +|---|---| +| RustySNES | pass | +| Mesen2 | **pass** — its failing set is `F1.03` and `F1.10`, nothing else | +| ares | **pass** (status `$01`) | +| snes9x | **FAIL code 1** — and it is documented, the *first* entry in `SNES9X_KNOWN_FAILURES` | + +So the outlier is **snes9x**, not Mesen2; the failure is expected and already explained; and there is +no unexplained disagreement on this row at all. The old text additionally asserted "`A2.10` is not +among snes9x's 14 known divergences" — it is, at `crossval.sh:80`. + +**How it went wrong.** `crossval.sh` reported `Mesen2: 1 failing test(s)`, and index 11 was read off +a *different host's* output and attributed to Mesen2. Neither number was checked against a per-row +measurement, because none existed: `mesen_crossval.lua` reports only an exit code and `emu.log` does +not reach any findable file here. `scripts/accuracysnes/mesen_failing_set_probe.lua` now does the +measurement, and gives the identical set on six consecutive runs. + +**This is the second published claim in one day undone by keying on catalogue indices instead of row +names**, the other being `MESEN2_KNOWN_FAILURES`'s own comment. An index is only meaningful next to +the host that produced it and the catalogue that was current — and neither travels with the number. + +**What survives.** The ares host was built to settle this row, and its stated motivation was +therefore partly wrong. It still earned itself several times over: it found a real bug in ares' +SMP (`$F1`'s timer-2 counter reset is negated), which needed a third opinion to see at all. #### The Mesen oracle — the "14 of 335" reading does not reproduce diff --git a/scripts/accuracysnes/ares_host/README.md b/scripts/accuracysnes/ares_host/README.md index accbcf9b..4dc9f520 100644 --- a/scripts/accuracysnes/ares_host/README.md +++ b/scripts/accuracysnes/ares_host/README.md @@ -25,9 +25,13 @@ Several findings sat at **2 versus 1** with no way to break the tie, because thi provenance rule counts ares and bsnes as **one** reference — so "RustySNES and snes9x against Mesen2" is only 2-vs-1 if ares is not already on RustySNES's side, and nobody could check. -**The first thing it settled: `A2.10` ("PEI does not page-wrap").** ares **passes** it (catalogue -index 11, status `$01`). With RustySNES and snes9x also passing, that is **3 against 1 with Mesen2 -the outlier**, and the row comes off the "unexplained, needs a fourth opinion" list. +**The row it was built to settle did not need settling.** `A2.10` was recorded as "Mesen2 fails it, +RustySNES and snes9x pass". Measured: ares passes it, **Mesen2 passes it**, RustySNES passes it, and +**snes9x fails it** — where it is documented as the first entry in `SNES9X_KNOWN_FAILURES`. The +original reading took a catalogue index off one host's output and attributed it to another. Recorded +here because it is this host's stated motivation, and the motivation was wrong. + +**What it did earn:** the first bug AccuracySNES has found in a *reference* emulator — see below. ## The five rows ares disagrees with the cart about diff --git a/scripts/accuracysnes/crossval.sh b/scripts/accuracysnes/crossval.sh index b9c875b2..455e3f2e 100755 --- a/scripts/accuracysnes/crossval.sh +++ b/scripts/accuracysnes/crossval.sh @@ -219,6 +219,45 @@ MESEN2_KNOWN_FAILURES=2 # NTSC constant because the two genuinely differ, and a shared constant would hide it. MESEN2_PAL_KNOWN_FAILURES=1 +# --- ares' own known divergences ----------------------------------------------------------------- +# +# ares is the THIRD reference, added 2026-08-01 (`scripts/accuracysnes/ares_host/`). It matters +# because this project's provenance rule counts ares and bsnes as ONE reference — so any row where +# RustySNES and snes9x face Mesen2 is only 2-vs-1 if ares is not already on RustySNES's side, and +# before this there was no way to check. +# +# Five rows, and THREE of them are rows snes9x already fails for reasons documented above — ares is +# corroborating snes9x there, not standing alone: +# +# C7.05 code 1 | Range Over's set dot. snes9x fails it too, on a DIFFERENT code (2), so the two +# | dissenters do not even agree with each other about how it is wrong. +# C7.10 code 1 | The Uniracers OAM-address takeover. snes9x fails it identically; Mesen2 models +# | it and RustySNES does too. 2-vs-2. +# F1.10 code 2 | The auto-read start race. snes9x, Mesen2 AND ares all fail it — see the F1.10 +# | entry under MESEN2_KNOWN_FAILURES for why that leaves RustySNES passing alone. +# +# The other two share ONE cause, and it is a bug in ares rather than a disagreement: +# +# E3.06 code 2 | ares' `$F1` (CONTROL) handler NEGATES the timer-2 counter reset: +# E8.02 code 3 | if(timer0.enable.raise(data.bit(0))) { timer0.stage2 = 0; ... } +# | if(timer1.enable.raise(data.bit(1))) { timer1.stage2 = 0; ... } +# | if(!timer2.enable.raise(data.bit(2))) { timer2.stage2 = 0; ... } +# | ^ negated, and only here (ares/sfc/smp/io.cpp) +# | `raise()` is true on a 0->1 transition, so timers 0 and 1 reset ON a raise -- +# | the documented behaviour -- while timer 2 resets on anything EXCEPT one. Both +# | rows enable timer 2 via `$F1`, run an interval, then write `$F1` again to STOP +# | it, and that stopping write has already zeroed T2OUT. Both read 0 where +# | RustySNES reads 10 and ~7. ares is internally inconsistent here, which is the +# | strongest evidence available that it is a stray `!`: its own timers 0 and 1 do +# | the un-negated version, and RustySNES, snes9x and Mesen2 all treat the three +# | identically. The first bug this cart has found in a REFERENCE emulator. +ARES_KNOWN_FAILURES=5 + +# Where the built host lives. Not built by this script: it is a C++ link against ares' static libs +# and takes minutes, so it is opt-in via `scripts/accuracysnes/ares_host/build.sh` and this block +# skips cleanly when the binary is absent — the same shape as the snes9x and Mesen2 blocks. +ARES_HOST=${ARES_HOST:-${TMPDIR:-/tmp}/ares_host} + # --- snes9x, via the libretro host -------------------------------------------------------------- if [[ -f $SNES9X ]]; then cc -O2 -o "$HOST" scripts/accuracysnes/libretro_crossval.c -ldl || exit 1 @@ -262,6 +301,30 @@ else echo "skip Mesen2: build it with 'make -C ref-proj/Mesen2'" >&2 fi +# --- ares, via the headless host ------------------------------------------------------------------ +if [[ -x $ARES_HOST ]]; then + echo "=== ares (headless host) ===" + # The host prints the results block on stdout; its failing count is derived here rather than + # returned as an exit code, because a status byte is `even = FAIL` and that is worth reading in + # one place rather than encoding twice. + # Parity off the LAST HEX DIGIT, not arithmetic on the string: POSIX awk does not parse "0x01", + # so `("0x" $3) % 2` is 0 for every byte and counts all 337 non-skipped rows as failures. That is + # a gate reporting catastrophe out of a parsing bug, which is worth the comment. + # $FF is SKIP and $00 is NOT-RUN; both are excluded, matching how the other hosts count. + n=$("$ARES_HOST" "$ROM" 900 2>/dev/null | + awk '$1 == "status" && $3 != "ff" && $3 != "00" && $3 ~ /[02468aceACE]$/ { c++ } + END { print c + 0 }') + if [[ $n -eq $ARES_KNOWN_FAILURES ]]; then + echo "ares: OK ($n known divergence(s) — see ARES_KNOWN_FAILURES above)" + else + echo "ares: $n failing test(s), expected $ARES_KNOWN_FAILURES" >&2 + rc=1 + fi + ran=$((ran + 1)) +else + echo "skip ares: build it with 'bash scripts/accuracysnes/ares_host/build.sh'" >&2 +fi + # --- the PAL image ------------------------------------------------------------------------------ # # The same battery at PAL timing. The two images differ in one header byte, so this is the cheapest