[Eval] Measure option-order sensitivity of each decision backend (flip rate under reversed/shuffled options) #90

Why

A decision should not depend on where an option sits in the list. Two papers from this month say this is not a given for System-1 backends:

• Fast Models, Slow Evidence (arXiv:2610.02267), a paired Laya vs Jev evaluation with released cases and raw outputs, reversed only the option order on 1,000 cases. Laya changed 30.3% of its decisions (53% with 60 to 77 labels), Jev 1.8%. In their routing track, Laya picked the same option 58.5% of the time in the original order and 82.3% reversed: "its preference follows position, not content". Their advice for Laya is to "fix the option order or vote over permutations".
• Jeva (arXiv:2610.18773) lists permutation invariance as one of four structural requirements for a System-1 model and reports no decision flips under any tested permutation. SearchJev (arXiv:2610.05107) permutes options during training "to reduce position bias".

System1-Agents never measures this. Each front builds its options in one deterministic order per state, and nothing re-asks with a different order. The ticket-router evals shuffle tickets across seeds (evals/ticket_router/compare_framings.py), not options. So a backend that is order-sensitive shows up as "accuracy", and a page whose element order changes could flip a browser step with no change in content. s1a probe and the recorded tick requests already have everything needed to check it offline.

Proposed scope (one PR)

1. Add an option-order check to the fit probe: s1a probe <cases.jsonl> --model <m> --order reverse|shuffle:N. It asks each case in the original order, then again with the option dict reordered (same keys, same texts, same state and rules). It reports:
   • the flip rate (share of cases whose chosen key changes),
   • accuracy in each order,
   • per-case rows for the flips.
2. Add the same check as a small eval over saved decisions: for an existing run directory, re-ask each recorded choice request with its options reversed and report the flip rate per backend. Ticket-router and browser tick records are the first targets, since both keep the full request. If replay inputs are incomplete for a front, say so instead of guessing.
3. Document the result in docs/capabilities.md as a separate robustness column/row per backend and hardware, with the date, checkpoint and case count. Do not fold it into accuracy.

Acceptance criteria

• Offline unit tests with a deliberately position-biased fake backend (always picks the first offered key) show a 100% flip rate under reverse, and a content-based fake shows 0%.
• Reordering keeps keys and texts byte-identical. Only the dict order changes, and the records show both orders.
• One real run on at least one backend (laya on CPU is enough) over the bundled ticket-router set or a probe case file, with the command, checkpoint, device and date recorded.
• Exit codes for s1a probe stay unchanged when --order is not passed.

Out of scope

• Mitigation (permutation voting, canonical sorting of options inside a front). Propose it as a follow-up once the flip rates are known, because voting multiplies decision latency.
• Wording sensitivity (the same paper reports a 5 to 15 pp band). It's worth doing, but it's a separate check.

References

• Fast Models, Slow Evidence: https://arxiv.org/abs/2610.02267 (data and code: https://github.com/David-DL-Space/sys1-eval)
• Jeva: System One Models from First Principles: https://arxiv.org/abs/2610.18773
• SearchJev: https://arxiv.org/abs/2610.05107
• Background on option-order bias in LLM selectors: Zheng et al., "Large Language Models Are Not Robust Multiple Choice Selectors" (ICLR 2024), https://arxiv.org/abs/2309.03882; Pezeshkpour & Hruschka (Findings of NAACL 2024), https://arxiv.org/abs/2308.11483
• Related: [Eval] Add repeatable browser and desktop task suites #13 (task suites), [Benchmark] Re-score public231 evidence with complete denominators #68 / [Feat] Re-score saved public231 decisions #69 (public231 re-scoring), [Decisions] Make confidence mean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin) #89 (confidence semantics)