Skip to content

feat(accuracysnes): ares as a third reference — and retract the A2.10 finding #309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
doublegate merged 1 commit into from
Aug 1, 2026
+129 −38
Merged

feat(accuracysnes): ares as a third reference — and retract the A2.10 finding #309

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **ares is wired into `crossval.sh` as a third reference.** `cross-validation: 3 reference(s) agree
with the cart`, with `ARES_KNOWN_FAILURES=5` carrying the same per-row rationale the other two
constants do: `C7.05`/`C7.10`/`F1.10` are rows snes9x already fails (ares corroborating it, not
standing alone), and `E3.06`/`E8.02` cite the ares `$F1` timer-2 reset bug.

The block is **opt-in and skips cleanly**: the host is a C++ link against ares' static libraries
and takes minutes, so `crossval.sh` looks for a pre-built binary at `$ARES_HOST` and prints
`skip ares: build it with 'bash scripts/accuracysnes/ares_host/build.sh'` when it is absent —
the same shape as the snes9x and Mesen2 blocks, and verified in both states.

One bug caught on the way, worth recording because of how it presented: counting the failing rows
with `("0x" $3) % 2 == 0` reported **337 failures**. POSIX awk does not parse `"0x01"`, so the
expression is `0 % 2` for every byte and every non-skipped row counted as a failure. A gate
reporting catastrophe out of a parsing bug is a specific kind of dangerous, so the fix matches on
the last hex digit and says why in a comment.

- **AccuracySNES found a bug in ares — the first time it has found one in a reference emulator.**
`E3.06` and `E8.02` were the two divergences the third opinion could not attribute to a known
snes9x failure, and both read **0** from every timer-2 slot they record. One mechanism explains
Expand Down Expand Up @@ -63,9 +79,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
magic ACSN / done a5 / count 338 / passed 299 / failed 5 / skipped 1 / golden 33
```

Reproducible across runs. **ares passes `A2.10`** ("PEI does not page-wrap", catalogue index 11,
status `$01`), so with RustySNES and snes9x it is **3 against 1 with Mesen2 the outlier**, and the
row comes off the "unexplained, needs a fourth opinion" list it has been on since the oracle fix.
Reproducible across runs. ares passes `A2.10` (catalogue index 11, status `$01`).

**RETRACTION.** This entry first said that made it "3 against 1 with Mesen2 the outlier". It does
not, because **Mesen2 passes `A2.10` too** — measured with `mesen_failing_set_probe.lua`, its
failing set is `F1.03` and `F1.10` and nothing else, identical on six runs. The host that fails
`A2.10` is **snes9x**, where it is documented as the *first* entry in `SNES9X_KNOWN_FAILURES`. So
there was never a disagreement on this row: the original reading took an index off one host's
output and attributed it to another. The ares host was built to settle a row that did not need
settling — and still earned itself, by finding a real ares bug that needed a third opinion to see.

**Five rows where ares disagrees with the cart** — and **three of them are rows snes9x already
fails**, which the tally alone does not show. `C7.10` and `F1.10` become **2-vs-2** (RustySNES +
Expand Down
66 changes: 34 additions & 32 deletions docs/accuracysnes-plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -839,38 +839,40 @@ cannot increment between two reads six cycles apart. Unless a distinguishing obs
first, this is the shape [[accuracysnes-vacuous-test-trap]] exists to prevent — design the observable
before writing the row.

**Before authoring any of them,** settle `A2.10` above. It is the only unexplained reference
disagreement in the battery, and a row batch authored while one row is unexplained risks attributing
a new failure to the wrong cause.

#### First arbitrated finding: `A2.10` — Mesen2 is the outlier, and ares makes it 3-vs-1

The moment the oracle started working it produced a result the project has never been able to see.
`crossval.sh` reports `Mesen2: 1 failing test(s)`, catalogue index **11 = cart `A2.10`, "PEI does not
page-wrap"** (scored; source: WDC datasheet + superfamicom.org addressing notes; verdict slot
`$7EF02B`).

RustySNES **passes** it and snes9x passes it too — `A2.10` is not among snes9x's 14 known
divergences. So this is **2-vs-1 with Mesen2 as the outlier**, which inverts the usual shape: the
standing heuristic warns that *RustySNES failing alone* means a real bug, and that is not what this
is.

**SETTLED 2026-08-01 — ares passes it, so this is 3-vs-1.** The third opinion the paragraph below
asked for now exists: `scripts/accuracysnes/ares_host/` runs the battery under ares
(`magic ACSN`, `done a5`, 338 tests, reproducible across runs) and reports catalogue index 11 as
status `$01` — **pass**. With RustySNES and snes9x that is three implementations against Mesen2, and
the caveat that held this open — a harness bug upstream of an implementation produces the same
signature, as the `$F8`/`$F9` retraction proved — is answered: the fourth host is independent of all
three and agrees.

`A2.10` can therefore become a `MESEN2_KNOWN_FAILURES` entry whenever the count is next revised. It
is not being added in the same change that established it, because `MESEN2_KNOWN_FAILURES` currently
reads 2 and matches; changing the constant and the reasoning at once would leave nothing to check
the change against.

*The original reasoning, kept because it is why the row waited:* recording it as a known-good Mesen2
divergence without a third opinion would have been exactly the mistake the `$F8`/`$F9` retraction
already cost this project once.
**~~Before authoring any of them, settle `A2.10` above.~~** Retracted with the section below: there
was never an `A2.10` disagreement to settle. The battery has **no** unexplained reference
disagreement — every failing row on every host is now measured and attributed.

#### RETRACTED: the `A2.10` "arbitrated finding" was a host mix-up

This section used to read *"`A2.10` — Mesen2 is the outlier, 2-vs-1"*, and called it the only
unexplained reference disagreement in the battery. **It was wrong in both directions**, and measuring
it took a probe that did not exist until 2026-08-01.

| | `A2.10` (catalogue index 11) |
|---|---|
| RustySNES | pass |
| Mesen2 | **pass** — its failing set is `F1.03` and `F1.10`, nothing else |
| ares | **pass** (status `$01`) |
| snes9x | **FAIL code 1** — and it is documented, the *first* entry in `SNES9X_KNOWN_FAILURES` |

So the outlier is **snes9x**, not Mesen2; the failure is expected and already explained; and there is
no unexplained disagreement on this row at all. The old text additionally asserted "`A2.10` is not
among snes9x's 14 known divergences" — it is, at `crossval.sh:80`.

**How it went wrong.** `crossval.sh` reported `Mesen2: 1 failing test(s)`, and index 11 was read off
a *different host's* output and attributed to Mesen2. Neither number was checked against a per-row
measurement, because none existed: `mesen_crossval.lua` reports only an exit code and `emu.log` does
not reach any findable file here. `scripts/accuracysnes/mesen_failing_set_probe.lua` now does the
measurement, and gives the identical set on six consecutive runs.

**This is the second published claim in one day undone by keying on catalogue indices instead of row
names**, the other being `MESEN2_KNOWN_FAILURES`'s own comment. An index is only meaningful next to
the host that produced it and the catalogue that was current — and neither travels with the number.

**What survives.** The ares host was built to settle this row, and its stated motivation was
therefore partly wrong. It still earned itself several times over: it found a real bug in ares'
SMP (`$F1`'s timer-2 counter reset is negated), which needed a third opinion to see at all.

#### The Mesen oracle — the "14 of 335" reading does not reproduce

Expand Down
10 changes: 7 additions & 3 deletions scripts/accuracysnes/ares_host/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,13 @@ Several findings sat at **2 versus 1** with no way to break the tie, because thi
provenance rule counts ares and bsnes as **one** reference — so "RustySNES and snes9x against
Mesen2" is only 2-vs-1 if ares is not already on RustySNES's side, and nobody could check.

**The first thing it settled: `A2.10` ("PEI does not page-wrap").** ares **passes** it (catalogue
index 11, status `$01`). With RustySNES and snes9x also passing, that is **3 against 1 with Mesen2
the outlier**, and the row comes off the "unexplained, needs a fourth opinion" list.
**The row it was built to settle did not need settling.** `A2.10` was recorded as "Mesen2 fails it,
RustySNES and snes9x pass". Measured: ares passes it, **Mesen2 passes it**, RustySNES passes it, and
**snes9x fails it** — where it is documented as the first entry in `SNES9X_KNOWN_FAILURES`. The
original reading took a catalogue index off one host's output and attributed it to another. Recorded
here because it is this host's stated motivation, and the motivation was wrong.

**What it did earn:** the first bug AccuracySNES has found in a *reference* emulator — see below.

## The five rows ares disagrees with the cart about

Expand Down
63 changes: 63 additions & 0 deletions scripts/accuracysnes/crossval.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,45 @@ MESEN2_KNOWN_FAILURES=2
# NTSC constant because the two genuinely differ, and a shared constant would hide it.
MESEN2_PAL_KNOWN_FAILURES=1

# --- ares' own known divergences -----------------------------------------------------------------
#
# ares is the THIRD reference, added 2026-08-01 (`scripts/accuracysnes/ares_host/`). It matters
# because this project's provenance rule counts ares and bsnes as ONE reference — so any row where
# RustySNES and snes9x face Mesen2 is only 2-vs-1 if ares is not already on RustySNES's side, and
# before this there was no way to check.
Comment on lines +222 to +227
#
# Five rows, and THREE of them are rows snes9x already fails for reasons documented above — ares is
# corroborating snes9x there, not standing alone:
#
# C7.05 code 1 | Range Over's set dot. snes9x fails it too, on a DIFFERENT code (2), so the two
# | dissenters do not even agree with each other about how it is wrong.
# C7.10 code 1 | The Uniracers OAM-address takeover. snes9x fails it identically; Mesen2 models
# | it and RustySNES does too. 2-vs-2.
# F1.10 code 2 | The auto-read start race. snes9x, Mesen2 AND ares all fail it — see the F1.10
# | entry under MESEN2_KNOWN_FAILURES for why that leaves RustySNES passing alone.
#
# The other two share ONE cause, and it is a bug in ares rather than a disagreement:
#
# E3.06 code 2 | ares' `$F1` (CONTROL) handler NEGATES the timer-2 counter reset:
# E8.02 code 3 | if(timer0.enable.raise(data.bit(0))) { timer0.stage2 = 0; ... }
# | if(timer1.enable.raise(data.bit(1))) { timer1.stage2 = 0; ... }
# | if(!timer2.enable.raise(data.bit(2))) { timer2.stage2 = 0; ... }
# | ^ negated, and only here (ares/sfc/smp/io.cpp)
# | `raise()` is true on a 0->1 transition, so timers 0 and 1 reset ON a raise --
# | the documented behaviour -- while timer 2 resets on anything EXCEPT one. Both
# | rows enable timer 2 via `$F1`, run an interval, then write `$F1` again to STOP
# | it, and that stopping write has already zeroed T2OUT. Both read 0 where
# | RustySNES reads 10 and ~7. ares is internally inconsistent here, which is the
# | strongest evidence available that it is a stray `!`: its own timers 0 and 1 do
# | the un-negated version, and RustySNES, snes9x and Mesen2 all treat the three
# | identically. The first bug this cart has found in a REFERENCE emulator.
ARES_KNOWN_FAILURES=5

Comment on lines +254 to +255

@coderabbitai coderabbitai Bot Aug 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Validate the complete ares result and exact failure rows.

The block accepts ares when n == ARES_KNOWN_FAILURES, but n comes from a loose suffix match. Five lines such as status 0 bogus2, five failures at unexpected indices, or truncated output with five matching status lines produce n=5 and print ares: OK. The block also ignores the result markers, magic, done, declared count, indexed row set, and the host exit status. If a failure moves from a documented row such as A2.10 while the total remains five, the gate still passes.

Capture the host output. Check the host exit status and the complete ACCURACYSNES result block. Compare the declared status count and the exact (row, status) set for the five documented ares divergences. Store those expectations as data, not only comments.

As per path instructions, the cross-validation harness must not accept a loosely parsed verdict that can make a reference appear to agree.

Also applies to: 314-322

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/accuracysnes/crossval.sh` around lines 254 - 255, Replace the loose
ARES_KNOWN_FAILURES count check in the ares validation block with structured
validation of captured host output: require a successful host exit status, a
complete ACCURACYSNES result block with valid magic and done markers, and a
declared count of five matching the exact documented (row, status) pairs. Store
those expected pairs as data and reject missing, extra, reordered, or
unexpected-index failures, as well as truncated or malformed output.

Source: Path instructions

# Where the built host lives. Not built by this script: it is a C++ link against ares' static libs
# and takes minutes, so it is opt-in via `scripts/accuracysnes/ares_host/build.sh` and this block
# skips cleanly when the binary is absent — the same shape as the snes9x and Mesen2 blocks.
ARES_HOST=${ARES_HOST:-${TMPDIR:-/tmp}/ares_host}

# --- snes9x, via the libretro host --------------------------------------------------------------
if [[ -f $SNES9X ]]; then
cc -O2 -o "$HOST" scripts/accuracysnes/libretro_crossval.c -ldl || exit 1
Expand Down Expand Up @@ -262,6 +301,30 @@ else
echo "skip Mesen2: build it with 'make -C ref-proj/Mesen2'" >&2
fi

# --- ares, via the headless host ------------------------------------------------------------------
if [[ -x $ARES_HOST ]]; then
echo "=== ares (headless host) ==="
# The host prints the results block on stdout; its failing count is derived here rather than
# returned as an exit code, because a status byte is `even = FAIL` and that is worth reading in
# one place rather than encoding twice.
# Parity off the LAST HEX DIGIT, not arithmetic on the string: POSIX awk does not parse "0x01",
# so `("0x" $3) % 2` is 0 for every byte and counts all 337 non-skipped rows as failures. That is
# a gate reporting catastrophe out of a parsing bug, which is worth the comment.
# $FF is SKIP and $00 is NOT-RUN; both are excluded, matching how the other hosts count.
n=$("$ARES_HOST" "$ROM" 900 2>/dev/null |
awk '$1 == "status" && $3 != "ff" && $3 != "00" && $3 ~ /[02468aceACE]$/ { c++ }
END { print c + 0 }')
if [[ $n -eq $ARES_KNOWN_FAILURES ]]; then
echo "ares: OK ($n known divergence(s) — see ARES_KNOWN_FAILURES above)"
else
echo "ares: $n failing test(s), expected $ARES_KNOWN_FAILURES" >&2
rc=1
fi
ran=$((ran + 1))
else
echo "skip ares: build it with 'bash scripts/accuracysnes/ares_host/build.sh'" >&2
fi

# --- the PAL image ------------------------------------------------------------------------------
#
# The same battery at PAL timing. The two images differ in one header byte, so this is the cheapest
Expand Down
Loading