[Birfy]

Closes #90

Required human contributor answers

Per the repository owner's standing instruction (overriding the human-answer gate from #71 for these maintenance runs), the intent answers below are filled from the issue text.

1. What outcome are you trying to achieve?

   Answer: Know whether a decision backend's answer depends on where an option sits in the list, measured separately from accuracy ([Eval] Measure option-order sensitivity of each decision backend (flip rate under reversed/shuffled options) #90).

2. What is the highest-leverage variable for that outcome?

   Answer: The option-order flip rate per backend: the share of cases whose chosen key changes when only the option dict order changes.

3. How will this work move that variable rather than create busywork?

   Answer: It adds the measurement to the existing fit probe (s1a probe --order) so any case file and any backend can be checked offline, and records the first real number (LAYA, CPU).

4. Who is the customer or user, and what do they actually need?

   Answer: People choosing a decision backend for a front (and reviewers of capability claims); they need to see order sensitivity before trusting "accuracy", as arXiv:2610.02267 reports for Laya.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale (or explicit source reference): Yes; the probe has no way to re-ask with another option order today (issue #90, "System1-Agents never measures this").

For a bug fix: link actual observed failure evidence.

Evidence: Not a bug fix (new eval capability). Real-run evidence is under docs/results/option-order/2026-10-21-laya-cpu/.

Why

#90: nothing re-asks a decision with a different option order, so a position-biased backend looks like an accurate one. arXiv:2610.02267 reports Laya flipping 30.3% of decisions when only the option order is reversed (Jev 1.8%).

How

• s1a/probe.py: parse_order (reverse | shuffle:N, N ≤ 50), reorderings (same keys and texts, only the dict order changes; shuffles are seeded and avoid the identity order), run_order (each case in file order, then in every reordering; flip rate, accuracy per order, rows keeping every order asked), render_order.
• s1a/cli.py: s1a probe ... --order reverse|shuffle:N [--seed S] [--json PATH]. A bad --order is a usage error (exit 2) before any model loads. Verdict and exit code still come from the file order; without --order nothing changes.

What

• New flags on s1a probe: --order, --seed, --json.
• tests/test_probe_order.py (9 tests): a position-biased fake (always the first offered key) flips 100% under reverse/shuffle; a content-based fake flips 0%; request bodies show both orders with byte-identical keys/texts/state; seeded shuffles are reproducible; CLI output and exit code are unchanged without --order; bad --order → exit 2.
• Docs: docs/skills.md (fit-probe step), docs/capabilities.md (separate robustness row, not folded into accuracy), docs/results/option-order/2026-10-21-laya-cpu/ (README + raw JSON), CHANGELOG.md.
• Deferred: scope item 2 of [Eval] Measure option-order sensitivity of each decision backend (flip rate under reversed/shuffled options) #90 (re-asking recorded ticket-router/browser requests from a saved run directory). It isn't in the acceptance criteria and can be a follow-up issue; mitigation (permutation voting) stays out of scope as the issue says.

Verification

• uv run ruff format --check . && uv run ruff check . && uv run ty check: clean; uv lock --check and uv build ok
• uv run pytest -q: 1000 passed, 127 skipped, 211 subtests passed; scripts/smoke.sh: ok
• CHANGELOG.md and the docs say what the code does now

Real run (acceptance item 3): in-process LAYA convaiinnovations/laya@7b928d8, laya 0.3.20, CPU only (Linux x86_64, 8 vCPU Xeon), 2026-10-21, over evals/ticket_router/probe.jsonl (12 cases):

Command	Flips	Acc. file order	Acc. reordered
uv run s1a probe evals/ticket_router/probe.jsonl --model laya --order reverse	3/12 (25%)	9/12	6/12
uv run s1a probe evals/ticket_router/probe.jsonl --model laya --order shuffle:5 --seed 0	3/12 (25%)	9/12	39/60

Demo / evidence

• Video link, or exemption/blocker with follow-up: N/A: a CLI measurement, with no agent task or UI. The raw JSON records are the evidence.
• Application/task → agent actions → independently checked result: s1a probe --order over the ticket-router probe cases; the per-case orders and choices are in docs/results/option-order/2026-10-21-laya-cpu/*.json.
• System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: Agents branch head on main 5693f51; no Omni (in-process laya 0.3.20); checkpoint convaiinnovations/laya@7b928d828b7b0e022f929d9bd2e44165aa270148; CPU.

[Birfy]

Reviewed at 6ce0129 (own PR, so a comment rather than an approval). CI is green.

The change does what #90 asks and stays out of the way when --order is not given: parse_order runs before any model loads, the verdict and exit code still come from the file order, and reordered dicts keep keys and texts identical. Tests cover parsing, the identity-avoiding shuffle and the flip count. Ready for a human approval from my side.

Two non-blocking notes:

1. A flip is not always position bias. A backend that is not deterministic (sampling, a served model behind a load balancer, GPU non-determinism) will show "flips" even if the order never changed. A cheap control would make the number interpretable: an --order same:N (or always one extra ask in file order) that reports the same-order disagreement rate next to the flip rate. Could be a follow-up; worth a sentence in docs/capabilities.md until then.
2. --seed and --json are silently ignored without --order. Either reject them (usage error, exit 2, like a bad --order) or say so in their help text.

[Birfy]

Pushed 843d5f8 for both review notes:

1. --order same:N is the non-determinism control: N more asks in file order, reported with the same flip/accuracy lines, so a reordered flip rate can be read against the backend's own same-order disagreement. Documented in docs/skills.md, the CHANGELOG, and a sentence on the LAYA CPU row in docs/capabilities.md (that run had no control).
2. --seed or --json without --order is now a usage error (exit 2, one line on stderr, before any model loads); --seed still defaults to 0 with --order.

Tests added for both; ruff, ty and the full suite (1003 passed) are green locally.

[hsliuustc0106]

Maintainer review (light-impact + self-review)

Tip: 843d5f891667d86a3fc527ed541d6224a0537bda · MERGEABLE · required checks green on this head.

Impact

• changed: probe.parse_order / reorderings / run_order / render_order; CLI --order/--seed/--json
• callers/dependents (first hop): cli.run_probe only; default probe path unchanged without --order
• related tests: tests/test_probe_order.py (bias, content, same:N control, usage errors)
• not expanded: saved-run replay (deferred #90 item 2)

Risk: low · Missing tests: none obvious for the new path · Regression: none without --order. Committed docs/results/option-order/... JSON is maintained evidence (has consumer/README), not one-off dump.

Self-review readiness: Intent from #90; CLI measurement demo N/A with raw JSON evidence is fine. Author nits (same:N, reject orphan --seed/--json) addressed on tip.

Verdict: APPROVE — eval capability is correct and opt-in.