diff --git a/CHANGELOG.md b/CHANGELOG.md index a538786a..12d88264 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **ares is wired into `crossval.sh` as a third reference.** `cross-validation: 3 reference(s) agree + with the cart`, with `ARES_KNOWN_FAILURES=5` carrying the same per-row rationale the other two + constants do: `C7.05`/`C7.10`/`F1.10` are rows snes9x already fails (ares corroborating it, not + standing alone), and `E3.06`/`E8.02` cite the ares `$F1` timer-2 reset bug. + + The block is **opt-in and skips cleanly**: the host is a C++ link against ares' static libraries + and takes minutes, so `crossval.sh` looks for a pre-built binary at `$ARES_HOST` and prints + `skip ares: build it with 'bash scripts/accuracysnes/ares_host/build.sh'` when it is absent — + the same shape as the snes9x and Mesen2 blocks, and verified in both states. + + One bug caught on the way, worth recording because of how it presented: counting the failing rows + with `("0x" $3) % 2 == 0` reported **337 failures**. POSIX awk does not parse `"0x01"`, so the + expression is `0 % 2` for every byte and every non-skipped row counted as a failure. A gate + reporting catastrophe out of a parsing bug is a specific kind of dangerous, so the fix matches on + the last hex digit and says why in a comment. + - **AccuracySNES found a bug in ares — the first time it has found one in a reference emulator.** `E3.06` and `E8.02` were the two divergences the third opinion could not attribute to a known snes9x failure, and both read **0** from every timer-2 slot they record. One mechanism explains @@ -63,9 +79,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 magic ACSN / done a5 / count 338 / passed 299 / failed 5 / skipped 1 / golden 33 ``` - Reproducible across runs. **ares passes `A2.10`** ("PEI does not page-wrap", catalogue index 11, - status `$01`), so with RustySNES and snes9x it is **3 against 1 with Mesen2 the outlier**, and the - row comes off the "unexplained, needs a fourth opinion" list it has been on since the oracle fix. + Reproducible across runs. ares passes `A2.10` (catalogue index 11, status `$01`). + + **RETRACTION.** This entry first said that made it "3 against 1 with Mesen2 the outlier". It does + not, because **Mesen2 passes `A2.10` too** — measured with `mesen_failing_set_probe.lua`, its + failing set is `F1.03` and `F1.10` and nothing else, identical on six runs. The host that fails + `A2.10` is **snes9x**, where it is documented as the *first* entry in `SNES9X_KNOWN_FAILURES`. So + there was never a disagreement on this row: the original reading took an index off one host's + output and attributed it to another. The ares host was built to settle a row that did not need + settling — and still earned itself, by finding a real ares bug that needed a third opinion to see. **Five rows where ares disagrees with the cart** — and **three of them are rows snes9x already fails**, which the tally alone does not show. `C7.10` and `F1.10` become **2-vs-2** (RustySNES + diff --git a/docs/accuracysnes-plan.md b/docs/accuracysnes-plan.md index dc3085b5..dbc526fc 100644 --- a/docs/accuracysnes-plan.md +++ b/docs/accuracysnes-plan.md @@ -839,38 +839,40 @@ cannot increment between two reads six cycles apart. Unless a distinguishing obs first, this is the shape [[accuracysnes-vacuous-test-trap]] exists to prevent — design the observable before writing the row. -**Before authoring any of them,** settle `A2.10` above. It is the only unexplained reference -disagreement in the battery, and a row batch authored while one row is unexplained risks attributing -a new failure to the wrong cause. - -#### First arbitrated finding: `A2.10` — Mesen2 is the outlier, and ares makes it 3-vs-1 - -The moment the oracle started working it produced a result the project has never been able to see. -`crossval.sh` reports `Mesen2: 1 failing test(s)`, catalogue index **11 = cart `A2.10`, "PEI does not -page-wrap"** (scored; source: WDC datasheet + superfamicom.org addressing notes; verdict slot -`$7EF02B`). - -RustySNES **passes** it and snes9x passes it too — `A2.10` is not among snes9x's 14 known -divergences. So this is **2-vs-1 with Mesen2 as the outlier**, which inverts the usual shape: the -standing heuristic warns that *RustySNES failing alone* means a real bug, and that is not what this -is. - -**SETTLED 2026-08-01 — ares passes it, so this is 3-vs-1.** The third opinion the paragraph below -asked for now exists: `scripts/accuracysnes/ares_host/` runs the battery under ares -(`magic ACSN`, `done a5`, 338 tests, reproducible across runs) and reports catalogue index 11 as -status `$01` — **pass**. With RustySNES and snes9x that is three implementations against Mesen2, and -the caveat that held this open — a harness bug upstream of an implementation produces the same -signature, as the `$F8`/`$F9` retraction proved — is answered: the fourth host is independent of all -three and agrees. - -`A2.10` can therefore become a `MESEN2_KNOWN_FAILURES` entry whenever the count is next revised. It -is not being added in the same change that established it, because `MESEN2_KNOWN_FAILURES` currently -reads 2 and matches; changing the constant and the reasoning at once would leave nothing to check -the change against. - -*The original reasoning, kept because it is why the row waited:* recording it as a known-good Mesen2 -divergence without a third opinion would have been exactly the mistake the `$F8`/`$F9` retraction -already cost this project once. +**~~Before authoring any of them, settle `A2.10` above.~~** Retracted with the section below: there +was never an `A2.10` disagreement to settle. The battery has **no** unexplained reference +disagreement — every failing row on every host is now measured and attributed. + +#### RETRACTED: the `A2.10` "arbitrated finding" was a host mix-up + +This section used to read *"`A2.10` — Mesen2 is the outlier, 2-vs-1"*, and called it the only +unexplained reference disagreement in the battery. **It was wrong in both directions**, and measuring +it took a probe that did not exist until 2026-08-01. + +| | `A2.10` (catalogue index 11) | +|---|---| +| RustySNES | pass | +| Mesen2 | **pass** — its failing set is `F1.03` and `F1.10`, nothing else | +| ares | **pass** (status `$01`) | +| snes9x | **FAIL code 1** — and it is documented, the *first* entry in `SNES9X_KNOWN_FAILURES` | + +So the outlier is **snes9x**, not Mesen2; the failure is expected and already explained; and there is +no unexplained disagreement on this row at all. The old text additionally asserted "`A2.10` is not +among snes9x's 14 known divergences" — it is, at `crossval.sh:80`. + +**How it went wrong.** `crossval.sh` reported `Mesen2: 1 failing test(s)`, and index 11 was read off +a *different host's* output and attributed to Mesen2. Neither number was checked against a per-row +measurement, because none existed: `mesen_crossval.lua` reports only an exit code and `emu.log` does +not reach any findable file here. `scripts/accuracysnes/mesen_failing_set_probe.lua` now does the +measurement, and gives the identical set on six consecutive runs. + +**This is the second published claim in one day undone by keying on catalogue indices instead of row +names**, the other being `MESEN2_KNOWN_FAILURES`'s own comment. An index is only meaningful next to +the host that produced it and the catalogue that was current — and neither travels with the number. + +**What survives.** The ares host was built to settle this row, and its stated motivation was +therefore partly wrong. It still earned itself several times over: it found a real bug in ares' +SMP (`$F1`'s timer-2 counter reset is negated), which needed a third opinion to see at all. #### The Mesen oracle — the "14 of 335" reading does not reproduce diff --git a/scripts/accuracysnes/ares_host/README.md b/scripts/accuracysnes/ares_host/README.md index accbcf9b..4dc9f520 100644 --- a/scripts/accuracysnes/ares_host/README.md +++ b/scripts/accuracysnes/ares_host/README.md @@ -25,9 +25,13 @@ Several findings sat at **2 versus 1** with no way to break the tie, because thi provenance rule counts ares and bsnes as **one** reference — so "RustySNES and snes9x against Mesen2" is only 2-vs-1 if ares is not already on RustySNES's side, and nobody could check. -**The first thing it settled: `A2.10` ("PEI does not page-wrap").** ares **passes** it (catalogue -index 11, status `$01`). With RustySNES and snes9x also passing, that is **3 against 1 with Mesen2 -the outlier**, and the row comes off the "unexplained, needs a fourth opinion" list. +**The row it was built to settle did not need settling.** `A2.10` was recorded as "Mesen2 fails it, +RustySNES and snes9x pass". Measured: ares passes it, **Mesen2 passes it**, RustySNES passes it, and +**snes9x fails it** — where it is documented as the first entry in `SNES9X_KNOWN_FAILURES`. The +original reading took a catalogue index off one host's output and attributed it to another. Recorded +here because it is this host's stated motivation, and the motivation was wrong. + +**What it did earn:** the first bug AccuracySNES has found in a *reference* emulator — see below. ## The five rows ares disagrees with the cart about diff --git a/scripts/accuracysnes/crossval.sh b/scripts/accuracysnes/crossval.sh index b9c875b2..455e3f2e 100755 --- a/scripts/accuracysnes/crossval.sh +++ b/scripts/accuracysnes/crossval.sh @@ -219,6 +219,45 @@ MESEN2_KNOWN_FAILURES=2 # NTSC constant because the two genuinely differ, and a shared constant would hide it. MESEN2_PAL_KNOWN_FAILURES=1 +# --- ares' own known divergences ----------------------------------------------------------------- +# +# ares is the THIRD reference, added 2026-08-01 (`scripts/accuracysnes/ares_host/`). It matters +# because this project's provenance rule counts ares and bsnes as ONE reference — so any row where +# RustySNES and snes9x face Mesen2 is only 2-vs-1 if ares is not already on RustySNES's side, and +# before this there was no way to check. +# +# Five rows, and THREE of them are rows snes9x already fails for reasons documented above — ares is +# corroborating snes9x there, not standing alone: +# +# C7.05 code 1 | Range Over's set dot. snes9x fails it too, on a DIFFERENT code (2), so the two +# | dissenters do not even agree with each other about how it is wrong. +# C7.10 code 1 | The Uniracers OAM-address takeover. snes9x fails it identically; Mesen2 models +# | it and RustySNES does too. 2-vs-2. +# F1.10 code 2 | The auto-read start race. snes9x, Mesen2 AND ares all fail it — see the F1.10 +# | entry under MESEN2_KNOWN_FAILURES for why that leaves RustySNES passing alone. +# +# The other two share ONE cause, and it is a bug in ares rather than a disagreement: +# +# E3.06 code 2 | ares' `$F1` (CONTROL) handler NEGATES the timer-2 counter reset: +# E8.02 code 3 | if(timer0.enable.raise(data.bit(0))) { timer0.stage2 = 0; ... } +# | if(timer1.enable.raise(data.bit(1))) { timer1.stage2 = 0; ... } +# | if(!timer2.enable.raise(data.bit(2))) { timer2.stage2 = 0; ... } +# | ^ negated, and only here (ares/sfc/smp/io.cpp) +# | `raise()` is true on a 0->1 transition, so timers 0 and 1 reset ON a raise -- +# | the documented behaviour -- while timer 2 resets on anything EXCEPT one. Both +# | rows enable timer 2 via `$F1`, run an interval, then write `$F1` again to STOP +# | it, and that stopping write has already zeroed T2OUT. Both read 0 where +# | RustySNES reads 10 and ~7. ares is internally inconsistent here, which is the +# | strongest evidence available that it is a stray `!`: its own timers 0 and 1 do +# | the un-negated version, and RustySNES, snes9x and Mesen2 all treat the three +# | identically. The first bug this cart has found in a REFERENCE emulator. +ARES_KNOWN_FAILURES=5 + +# Where the built host lives. Not built by this script: it is a C++ link against ares' static libs +# and takes minutes, so it is opt-in via `scripts/accuracysnes/ares_host/build.sh` and this block +# skips cleanly when the binary is absent — the same shape as the snes9x and Mesen2 blocks. +ARES_HOST=${ARES_HOST:-${TMPDIR:-/tmp}/ares_host} + # --- snes9x, via the libretro host -------------------------------------------------------------- if [[ -f $SNES9X ]]; then cc -O2 -o "$HOST" scripts/accuracysnes/libretro_crossval.c -ldl || exit 1 @@ -262,6 +301,30 @@ else echo "skip Mesen2: build it with 'make -C ref-proj/Mesen2'" >&2 fi +# --- ares, via the headless host ------------------------------------------------------------------ +if [[ -x $ARES_HOST ]]; then + echo "=== ares (headless host) ===" + # The host prints the results block on stdout; its failing count is derived here rather than + # returned as an exit code, because a status byte is `even = FAIL` and that is worth reading in + # one place rather than encoding twice. + # Parity off the LAST HEX DIGIT, not arithmetic on the string: POSIX awk does not parse "0x01", + # so `("0x" $3) % 2` is 0 for every byte and counts all 337 non-skipped rows as failures. That is + # a gate reporting catastrophe out of a parsing bug, which is worth the comment. + # $FF is SKIP and $00 is NOT-RUN; both are excluded, matching how the other hosts count. + n=$("$ARES_HOST" "$ROM" 900 2>/dev/null | + awk '$1 == "status" && $3 != "ff" && $3 != "00" && $3 ~ /[02468aceACE]$/ { c++ } + END { print c + 0 }') + if [[ $n -eq $ARES_KNOWN_FAILURES ]]; then + echo "ares: OK ($n known divergence(s) — see ARES_KNOWN_FAILURES above)" + else + echo "ares: $n failing test(s), expected $ARES_KNOWN_FAILURES" >&2 + rc=1 + fi + ran=$((ran + 1)) +else + echo "skip ares: build it with 'bash scripts/accuracysnes/ares_host/build.sh'" >&2 +fi + # --- the PAL image ------------------------------------------------------------------------------ # # The same battery at PAL timing. The two images differ in one header byte, so this is the cheapest