diff --git a/CHANGELOG.md b/CHANGELOG.md index fa5e7ff..236995b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ### Added +- `s1a probe --order reverse|shuffle:N` (with `--seed` and `--json PATH`): the option-order check. Every case is + asked in file order and again with the same keys and texts in another dict order; the probe reports the flip + rate, accuracy in each order and a row per case with every order asked. Verdict and exit code are unchanged + without `--order`. `--order same:N` is the control: N more asks in file order, so a backend that disagrees with + itself (sampling, a load-balanced server, GPU non-determinism) shows its own "flip" rate next to the reordered + one. `--seed` or `--json` without `--order` is a usage error (exit 2) rather than silently ignored. See #90. - Offline quality consumer for shared Omni public231 raw, retaining planned denominators and original scoring. - `s1a-mcp` tools carry titles and MCP behavior hints: `list_agents` and `decide` are read-only (so hosts with diff --git a/docs/capabilities.md b/docs/capabilities.md index 7ead6a4..7e882e0 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -27,6 +27,7 @@ These rows cite exact artifacts found in this checkout or clearly identify exter | Agents revision and artifact | Model / checkpoint and serving path | Input / question / scenario | Hardware actually identified | What the artifact establishes and does not establish | |---|---|---|---|---| +| Branch `issue-90-option-order-flip-rate` on `main` `5693f51` (#90), [`option-order/2026-10-21-laya-cpu`](results/option-order/2026-10-21-laya-cpu/README.md) | LAYA English `convaiinnovations/laya@7b928d8`, in-process `laya` 0.3.20 | Option-order robustness (not accuracy): `s1a probe --order reverse` and `--order shuffle:5 --seed 0` over the 12 ticket-router probe cases | Linux x86_64, 8 vCPU Intel Xeon; CPU, no CUDA | Flip rate 3/12 (25%) under both reverse and five shuffles; file-order accuracy 9/12, reordered 6/12 (reverse) and 39/60 (shuffles). One checkpoint, one case file, one machine; other backends and devices unmeasured. In-process LAYA on CPU is expected to be deterministic, but this run had no `--order same:N` control, so the same-order disagreement rate was not measured. | | Agents historical `served-laya` branch (report does not give a complete Agents SHA), [`evals/ticket_router/SERVED_LAYA.md`](../evals/ticket_router/SERVED_LAYA.md) and [`served_laya_records.json`](../evals/ticket_router/served_laya_records.json) | LAYA English `convaiinnovations/laya@55cf4c4`; in-process C-in, direct worker C-direct, and Omni `omni-jev` front C-front | Ticket router, 30-ticket set shuffled across three seeds; 90 decisions/configuration | Apple M1 Pro, 16 GB, macOS 26.1; MPS for served C-direct/C-front and in-process MPS for C-in | Records 63/90 correct for each configuration. Direct worker used compiled fp16 MPS; C-front routed through Omni; all (seed,ticket) pairs agreed across configs. Historical report, incomplete Agents source SHA; one application and checkpoint only, not broad quality evidence. | | Replay shaping at Agents `9f87fed`; separate live browser runs at `70227de` and `69ad3e5`; [`laya-browser-shaping/README.md`](results/laya-browser-shaping/README.md) and [`served-system1-omni/README.md`](results/laya-browser-shaping/served-system1-omni/README.md) | LAYA text worker through Omni `omni-jev`; Omni `ae86032cba2466f45f42c2ebdfadbcfa8c30eb9f`; worker checkpoint routing was not confirmed for every replay tick | Standalone `urllib` replay of 12 recorded Google Flights browser ticks, 45 questions; pre-shaped with Agents `laya_state()`/`laya_browser_question()`, sending requested 1536/1024 and default windows; separate live browser runs | Replay: Linux x86_64, 8 CPU cores, no GPU. Live run: Windows 11 CPU | The replay uses pre-shaped bodies and requested window fields, but it is not execution through Agents `ServedLayaModel`. The older worker ignored the requested window values and matched its default-window answers; per-tick checkpoint routing is uncertain. Separate live in-process browser attempts ended in early `DONE` and did not complete the task. These artifacts show input-shape and serving-request behavior, not a successful browser application. | | Agents `e64eb58e7fd94806edfa6206aa08e66f01bdae78`, external [PR review evidence release](https://github.com/QianCyrus/system1-agents/releases/tag/pr-review-evidence-20261007) | Served LAYA English `convaiinnovations/laya@55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`; Omni `99865743d27316fbe81362dc0f0e6de6fda86284`, `frontend.laya_mps` direct HTTP worker | Desktop document fixture; four decisions in one task: Save, Save, type, Save; independent file check | macOS 26.2 arm64; MPS; physical chip not reported | One historical fixture task succeeded on the final Save, after two ineffective Save choices; fresh file and exact content were independently checked. External archive also pins Laya and both repository revisions. One task only; not broad desktop capability. | diff --git a/docs/results/option-order/2026-10-21-laya-cpu/README.md b/docs/results/option-order/2026-10-21-laya-cpu/README.md new file mode 100644 index 0000000..9bd7b32 --- /dev/null +++ b/docs/results/option-order/2026-10-21-laya-cpu/README.md @@ -0,0 +1,25 @@ +# Option-order check: in-process LAYA on CPU, ticket-router probe cases (2026-10-21) + +First real run of `s1a probe --order` (#90). One backend, one case file, one machine; a robustness measurement, +not an accuracy claim and not a rate for other checkpoints or hardware. + +- Agents: this PR's branch `issue-90-option-order-flip-rate`, based on `main` `5693f51`. +- Model: `--model laya`, in-process `laya` 0.3.20, checkpoint `convaiinnovations/laya@7b928d828b7b0e022f929d9bd2e44165aa270148` + (English; laya warned that some shipped temperatures were clamped, so its own confidence is uncalibrated). +- Device: CPU only (torch 2.14.0, no CUDA device visible), Linux x86_64, 8 vCPU Intel Xeon, 15 GB RAM. +- Cases: `evals/ticket_router/probe.jsonl`, 12 cases, 5 options each. +- Date: 2026-10-21 (Asia/Shanghai). + +| Command | Flips | Accuracy, file order | Accuracy, reordered asks | Median ms | +|---|---|---|---|---| +| `uv run s1a probe evals/ticket_router/probe.jsonl --model laya --order reverse --json laya-reverse.json` | 3/12 (25%) | 9/12 (75%) | 6/12 (50%) | 420 | +| `uv run s1a probe evals/ticket_router/probe.jsonl --model laya --order shuffle:5 --seed 0 --json laya-shuffle5.json` | 3/12 (25%) | 9/12 (75%) | 39/60 (65%) | 247 | + +Under `reverse` the flips were "negated background" (returns → logistics), "background payment" (account → payment) +and "negation insufficient" (human → logistics); two of the three moved from a right answer to a wrong one. Under +five shuffles, "insufficient information" flipped instead of "negation insufficient". The JSON files keep every +order asked (`rows[].asks[].options`) and the key chosen in it. Latency is CPU wall time per decision on a shared +box and is not a performance claim. + +Not covered here: the saved-run replay over recorded ticket-router/browser requests (scope item 2 of #90), other +backends (Jev, CLM, served Laya), CUDA/MPS, and the multilingual checkpoint. diff --git a/docs/results/option-order/2026-10-21-laya-cpu/laya-reverse.json b/docs/results/option-order/2026-10-21-laya-cpu/laya-reverse.json new file mode 100644 index 0000000..424921e --- /dev/null +++ b/docs/results/option-order/2026-10-21-laya-cpu/laya-reverse.json @@ -0,0 +1,425 @@ +{ + "model": "laya", + "order": "reverse", + "seed": 0, + "rows": [ + { + "note": "ordinary logistics", + "choice": "logistics", + "ok": true, + "reordered_choices": [ + "logistics" + ], + "reordered_ok": 1, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 452 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 420 + } + ] + }, + { + "note": "negation", + "choice": "logistics", + "ok": true, + "reordered_choices": [ + "logistics" + ], + "reordered_ok": 1, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 400 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 395 + } + ] + }, + { + "note": "ordinary payment", + "choice": "payment", + "ok": true, + "reordered_choices": [ + "payment" + ], + "reordered_ok": 1, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "payment", + "ms": 381 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "payment", + "ms": 378 + } + ] + }, + { + "note": "background keyword", + "choice": "payment", + "ok": true, + "reordered_choices": [ + "payment" + ], + "reordered_ok": 1, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "payment", + "ms": 401 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "payment", + "ms": 267 + } + ] + }, + { + "note": "ordinary returns", + "choice": "returns", + "ok": true, + "reordered_choices": [ + "returns" + ], + "reordered_ok": 1, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "returns", + "ms": 285 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "returns", + "ms": 450 + } + ] + }, + { + "note": "negated background", + "choice": "returns", + "ok": true, + "reordered_choices": [ + "logistics" + ], + "reordered_ok": 0, + "flipped": true, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "returns", + "ms": 459 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 500 + } + ] + }, + { + "note": "ordinary account", + "choice": "account", + "ok": true, + "reordered_choices": [ + "account" + ], + "reordered_ok": 1, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "account", + "ms": 411 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "account", + "ms": 474 + } + ] + }, + { + "note": "background payment", + "choice": "account", + "ok": true, + "reordered_choices": [ + "payment" + ], + "reordered_ok": 0, + "flipped": true, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "account", + "ms": 415 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "payment", + "ms": 476 + } + ] + }, + { + "note": "independent intents", + "choice": "logistics", + "ok": false, + "reordered_choices": [ + "logistics" + ], + "reordered_ok": 0, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 416 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 471 + } + ] + }, + { + "note": "insufficient information", + "choice": "logistics", + "ok": false, + "reordered_choices": [ + "logistics" + ], + "reordered_ok": 0, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 452 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 431 + } + ] + }, + { + "note": "explicit human", + "choice": "logistics", + "ok": false, + "reordered_choices": [ + "logistics" + ], + "reordered_ok": 0, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 427 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 421 + } + ] + }, + { + "note": "negation insufficient", + "choice": "human", + "ok": true, + "reordered_choices": [ + "logistics" + ], + "reordered_ok": 0, + "flipped": true, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "human", + "ms": 418 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 430 + } + ] + } + ], + "cases": 12, + "flips": 3, + "flip_rate": 0.25, + "right": 9, + "accuracy": 0.75, + "reordered_asks": 12, + "reordered_right": 6, + "reordered_accuracy": 0.5, + "median_ms": 420, + "verdict": "not a decision-model task" +} diff --git a/docs/results/option-order/2026-10-21-laya-cpu/laya-shuffle5.json b/docs/results/option-order/2026-10-21-laya-cpu/laya-shuffle5.json new file mode 100644 index 0000000..f027659 --- /dev/null +++ b/docs/results/option-order/2026-10-21-laya-cpu/laya-shuffle5.json @@ -0,0 +1,1001 @@ +{ + "model": "laya", + "order": "shuffle:5", + "seed": 0, + "rows": [ + { + "note": "ordinary logistics", + "choice": "logistics", + "ok": true, + "reordered_choices": [ + "logistics", + "logistics", + "logistics", + "logistics", + "logistics" + ], + "reordered_ok": 5, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 326 + }, + { + "options": [ + "returns", + "payment", + "logistics", + "human", + "account" + ], + "choice": "logistics", + "ms": 292 + }, + { + "options": [ + "logistics", + "returns", + "payment", + "account", + "human" + ], + "choice": "logistics", + "ms": 298 + }, + { + "options": [ + "payment", + "logistics", + "human", + "returns", + "account" + ], + "choice": "logistics", + "ms": 245 + }, + { + "options": [ + "returns", + "logistics", + "account", + "payment", + "human" + ], + "choice": "logistics", + "ms": 284 + }, + { + "options": [ + "payment", + "human", + "account", + "returns", + "logistics" + ], + "choice": "logistics", + "ms": 307 + } + ] + }, + { + "note": "negation", + "choice": "logistics", + "ok": true, + "reordered_choices": [ + "logistics", + "logistics", + "logistics", + "logistics", + "logistics" + ], + "reordered_ok": 5, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 321 + }, + { + "options": [ + "payment", + "account", + "human", + "logistics", + "returns" + ], + "choice": "logistics", + "ms": 369 + }, + { + "options": [ + "payment", + "logistics", + "human", + "account", + "returns" + ], + "choice": "logistics", + "ms": 309 + }, + { + "options": [ + "human", + "logistics", + "payment", + "account", + "returns" + ], + "choice": "logistics", + "ms": 278 + }, + { + "options": [ + "logistics", + "returns", + "payment", + "account", + "human" + ], + "choice": "logistics", + "ms": 276 + }, + { + "options": [ + "returns", + "payment", + "account", + "human", + "logistics" + ], + "choice": "logistics", + "ms": 250 + } + ] + }, + { + "note": "ordinary payment", + "choice": "payment", + "ok": true, + "reordered_choices": [ + "payment", + "payment", + "payment", + "payment", + "payment" + ], + "reordered_ok": 5, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "payment", + "ms": 285 + }, + { + "options": [ + "returns", + "human", + "payment", + "account", + "logistics" + ], + "choice": "payment", + "ms": 250 + }, + { + "options": [ + "payment", + "human", + "account", + "logistics", + "returns" + ], + "choice": "payment", + "ms": 250 + }, + { + "options": [ + "logistics", + "account", + "returns", + "human", + "payment" + ], + "choice": "payment", + "ms": 247 + }, + { + "options": [ + "account", + "returns", + "payment", + "human", + "logistics" + ], + "choice": "payment", + "ms": 279 + }, + { + "options": [ + "human", + "payment", + "account", + "returns", + "logistics" + ], + "choice": "payment", + "ms": 247 + } + ] + }, + { + "note": "background keyword", + "choice": "payment", + "ok": true, + "reordered_choices": [ + "payment", + "payment", + "payment", + "payment", + "payment" + ], + "reordered_ok": 5, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "payment", + "ms": 240 + }, + { + "options": [ + "payment", + "human", + "account", + "returns", + "logistics" + ], + "choice": "payment", + "ms": 247 + }, + { + "options": [ + "account", + "logistics", + "payment", + "returns", + "human" + ], + "choice": "payment", + "ms": 309 + }, + { + "options": [ + "returns", + "logistics", + "payment", + "account", + "human" + ], + "choice": "payment", + "ms": 354 + }, + { + "options": [ + "account", + "human", + "logistics", + "payment", + "returns" + ], + "choice": "payment", + "ms": 234 + }, + { + "options": [ + "account", + "human", + "payment", + "returns", + "logistics" + ], + "choice": "payment", + "ms": 248 + } + ] + }, + { + "note": "ordinary returns", + "choice": "returns", + "ok": true, + "reordered_choices": [ + "returns", + "returns", + "returns", + "returns", + "returns" + ], + "reordered_ok": 5, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "returns", + "ms": 242 + }, + { + "options": [ + "account", + "returns", + "human", + "payment", + "logistics" + ], + "choice": "returns", + "ms": 215 + }, + { + "options": [ + "human", + "payment", + "returns", + "account", + "logistics" + ], + "choice": "returns", + "ms": 212 + }, + { + "options": [ + "returns", + "account", + "logistics", + "payment", + "human" + ], + "choice": "returns", + "ms": 216 + }, + { + "options": [ + "logistics", + "account", + "payment", + "returns", + "human" + ], + "choice": "returns", + "ms": 275 + }, + { + "options": [ + "human", + "account", + "payment", + "logistics", + "returns" + ], + "choice": "returns", + "ms": 265 + } + ] + }, + { + "note": "negated background", + "choice": "returns", + "ok": true, + "reordered_choices": [ + "returns", + "logistics", + "returns", + "logistics", + "logistics" + ], + "reordered_ok": 2, + "flipped": true, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "returns", + "ms": 225 + }, + { + "options": [ + "payment", + "human", + "logistics", + "returns", + "account" + ], + "choice": "returns", + "ms": 221 + }, + { + "options": [ + "payment", + "account", + "human", + "returns", + "logistics" + ], + "choice": "logistics", + "ms": 244 + }, + { + "options": [ + "logistics", + "human", + "account", + "payment", + "returns" + ], + "choice": "returns", + "ms": 340 + }, + { + "options": [ + "payment", + "returns", + "account", + "human", + "logistics" + ], + "choice": "logistics", + "ms": 232 + }, + { + "options": [ + "payment", + "returns", + "account", + "human", + "logistics" + ], + "choice": "logistics", + "ms": 215 + } + ] + }, + { + "note": "ordinary account", + "choice": "account", + "ok": true, + "reordered_choices": [ + "account", + "account", + "account", + "account", + "account" + ], + "reordered_ok": 5, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "account", + "ms": 240 + }, + { + "options": [ + "returns", + "account", + "human", + "logistics", + "payment" + ], + "choice": "account", + "ms": 238 + }, + { + "options": [ + "payment", + "human", + "account", + "logistics", + "returns" + ], + "choice": "account", + "ms": 210 + }, + { + "options": [ + "account", + "logistics", + "returns", + "human", + "payment" + ], + "choice": "account", + "ms": 219 + }, + { + "options": [ + "human", + "logistics", + "returns", + "payment", + "account" + ], + "choice": "account", + "ms": 266 + }, + { + "options": [ + "payment", + "human", + "returns", + "account", + "logistics" + ], + "choice": "account", + "ms": 264 + } + ] + }, + { + "note": "background payment", + "choice": "account", + "ok": true, + "reordered_choices": [ + "account", + "payment", + "account", + "payment", + "payment" + ], + "reordered_ok": 2, + "flipped": true, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "account", + "ms": 268 + }, + { + "options": [ + "payment", + "human", + "account", + "logistics", + "returns" + ], + "choice": "account", + "ms": 241 + }, + { + "options": [ + "account", + "logistics", + "payment", + "human", + "returns" + ], + "choice": "payment", + "ms": 307 + }, + { + "options": [ + "payment", + "returns", + "human", + "account", + "logistics" + ], + "choice": "account", + "ms": 244 + }, + { + "options": [ + "logistics", + "returns", + "human", + "payment", + "account" + ], + "choice": "payment", + "ms": 250 + }, + { + "options": [ + "human", + "logistics", + "returns", + "payment", + "account" + ], + "choice": "payment", + "ms": 219 + } + ] + }, + { + "note": "independent intents", + "choice": "logistics", + "ok": false, + "reordered_choices": [ + "logistics", + "logistics", + "logistics", + "logistics", + "logistics" + ], + "reordered_ok": 0, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 222 + }, + { + "options": [ + "returns", + "account", + "human", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 271 + }, + { + "options": [ + "account", + "payment", + "logistics", + "returns", + "human" + ], + "choice": "logistics", + "ms": 282 + }, + { + "options": [ + "account", + "returns", + "human", + "logistics", + "payment" + ], + "choice": "logistics", + "ms": 273 + }, + { + "options": [ + "logistics", + "payment", + "account", + "returns", + "human" + ], + "choice": "logistics", + "ms": 279 + }, + { + "options": [ + "payment", + "human", + "logistics", + "returns", + "account" + ], + "choice": "logistics", + "ms": 229 + } + ] + }, + { + "note": "insufficient information", + "choice": "logistics", + "ok": false, + "reordered_choices": [ + "logistics", + "logistics", + "returns", + "returns", + "logistics" + ], + "reordered_ok": 0, + "flipped": true, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 241 + }, + { + "options": [ + "returns", + "human", + "payment", + "logistics", + "account" + ], + "choice": "logistics", + "ms": 227 + }, + { + "options": [ + "payment", + "account", + "logistics", + "returns", + "human" + ], + "choice": "logistics", + "ms": 226 + }, + { + "options": [ + "logistics", + "payment", + "human", + "returns", + "account" + ], + "choice": "returns", + "ms": 236 + }, + { + "options": [ + "logistics", + "account", + "human", + "payment", + "returns" + ], + "choice": "returns", + "ms": 220 + }, + { + "options": [ + "payment", + "logistics", + "returns", + "human", + "account" + ], + "choice": "logistics", + "ms": 223 + } + ] + }, + { + "note": "explicit human", + "choice": "logistics", + "ok": false, + "reordered_choices": [ + "logistics", + "logistics", + "logistics", + "logistics", + "logistics" + ], + "reordered_ok": 0, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 259 + }, + { + "options": [ + "human", + "account", + "returns", + "payment", + "logistics" + ], + "choice": "logistics", + "ms": 237 + }, + { + "options": [ + "returns", + "account", + "logistics", + "human", + "payment" + ], + "choice": "logistics", + "ms": 260 + }, + { + "options": [ + "returns", + "payment", + "logistics", + "human", + "account" + ], + "choice": "logistics", + "ms": 235 + }, + { + "options": [ + "payment", + "logistics", + "returns", + "account", + "human" + ], + "choice": "logistics", + "ms": 262 + }, + { + "options": [ + "returns", + "human", + "logistics", + "account", + "payment" + ], + "choice": "logistics", + "ms": 324 + } + ] + }, + { + "note": "negation insufficient", + "choice": "human", + "ok": true, + "reordered_choices": [ + "human", + "human", + "human", + "human", + "human" + ], + "reordered_ok": 5, + "flipped": false, + "asks": [ + { + "options": [ + "logistics", + "payment", + "returns", + "account", + "human" + ], + "choice": "human", + "ms": 224 + }, + { + "options": [ + "logistics", + "human", + "returns", + "account", + "payment" + ], + "choice": "human", + "ms": 240 + }, + { + "options": [ + "returns", + "payment", + "account", + "logistics", + "human" + ], + "choice": "human", + "ms": 289 + }, + { + "options": [ + "account", + "logistics", + "payment", + "human", + "returns" + ], + "choice": "human", + "ms": 245 + }, + { + "options": [ + "human", + "logistics", + "returns", + "payment", + "account" + ], + "choice": "human", + "ms": 287 + }, + { + "options": [ + "account", + "returns", + "human", + "logistics", + "payment" + ], + "choice": "human", + "ms": 239 + } + ] + } + ], + "cases": 12, + "flips": 3, + "flip_rate": 0.25, + "right": 9, + "accuracy": 0.75, + "reordered_asks": 60, + "reordered_right": 39, + "reordered_accuracy": 0.65, + "median_ms": 247, + "verdict": "not a decision-model task" +} diff --git a/docs/skills.md b/docs/skills.md index cdc896d..58c12fb 100644 --- a/docs/skills.md +++ b/docs/skills.md @@ -108,6 +108,12 @@ It produces the module, its test and a row in the agents table, and stops at the 2. Fit probe before any code: 8 to 12 hand-written decisions through `s1a probe cases.jsonl`. Under 80 percent right, or one miss that needed deduction, the verdict is "not a decision-model task". Each case needs at least one option and one `accept` key from its `options`; invalid cases stop before any model call. + `s1a probe cases.jsonl --order reverse` (or `--order shuffle:N --seed S`) re-asks every case with the same + options in another dict order and prints the flip rate next to accuracy in each order; `--json PATH` keeps every + order asked. The verdict and exit code still come from the file order, and nothing changes without `--order`. + A backend whose answer follows position rather than content shows up here, not in accuracy (#90). A flip is + only position bias if the backend is deterministic: run `--order same:N` too, which re-asks in file order, and + read the reordered flip rate against that same-order disagreement rate. 3. Front: a tool loop for an environment that enumerates moves and scores, a browser policy for a page with visible controls, a rail for one question at a hook of a running agent. 4. Scaffold from the front's template under `s1a/agents/_templates/`, with the state-design rules from the skill's diff --git a/s1a/cli.py b/s1a/cli.py index 134d284..e0ee4a1 100644 --- a/s1a/cli.py +++ b/s1a/cli.py @@ -65,6 +65,18 @@ def parser() -> argparse.ArgumentParser: default="jev", help="who answers: jev, clm or laya-served over HTTP, laya and cua in process", ) + fit.add_argument( + "--order", + metavar="reverse|shuffle:N|same:N", + help=( + "also ask every case with its options reversed, in N seeded shuffles, or N more times in file order " + "(same:N, the non-determinism control); report the flip rate (#90)" + ), + ) + fit.add_argument("--seed", type=int, default=None, help="with --order shuffle:N: the shuffle seed (default 0)") + fit.add_argument( + "--json", type=Path, metavar="PATH", help="with --order: write the full summary, every order asked, here" + ) return build @@ -109,14 +121,30 @@ async def decide(args: argparse.Namespace) -> dict[str, Any]: return answer -async def run_probe(cases: Path, model_name: str) -> dict[str, Any]: +async def run_probe( + cases: Path, model_name: str, order: str | None = None, *, seed: int | None = None, json_path: Path | None = None +) -> dict[str, Any]: + """The fit probe; with ``order``, the option-order check over the same cases (verdict still from file order).""" + if order is None and (seed is not None or json_path is not None): + raise ValueError("--seed and --json only apply with --order") # a usage error, not silently ignored read = probe.read_cases(cases) + if order is not None: + probe.parse_order(order) # a bad --order is a usage error before any model loads + seed = 0 if seed is None else seed decision_model = build_model(model_name) try: - summary = await probe.run(read, decision_model) + if order is None: + summary = await probe.run(read, decision_model) + else: + summary = await probe.run_order(read, decision_model, order, seed=seed) finally: await decision_model.close() - print(probe.render(summary)) + if order is None: + print(probe.render(summary)) + else: + print(probe.render_order(summary)) + if json_path is not None: + json_path.write_text(json.dumps({"model": model_name, **summary}, ensure_ascii=False, indent=1) + "\n") return summary @@ -136,7 +164,10 @@ def main(argv: list[str]) -> int: case "decide": asyncio.run(decide(args)) case "probe": - return 0 if asyncio.run(run_probe(args.cases, args.model))["verdict"] == "fits" else 1 + summary = asyncio.run( + run_probe(args.cases, args.model, args.order, seed=args.seed, json_path=args.json) + ) + return 0 if summary["verdict"] == "fits" else 1 except (agents.UnknownAgent, ValueError) as exc: # JSONDecodeError is a ValueError print(exc, file=sys.stderr) return 2 diff --git a/s1a/probe.py b/s1a/probe.py index 40c328e..cd6576f 100644 --- a/s1a/probe.py +++ b/s1a/probe.py @@ -5,11 +5,17 @@ "note": "..."}``. ``accept`` lists every key that counts as right. A task fits when at least ``MIN_CASES`` cases were asked and at least ``FIT_THRESHOLD`` of them are right; a case that needs arithmetic or deduction to answer is a stop by itself, whatever the score. + +``run_order`` is the option-order check (#90): every case is asked in its file order, then again with the same option +dict reordered (``reverse``, or ``shuffle:N`` seeded permutations). Keys, texts, state and rules stay byte-identical; +only the dict order changes. It reports the flip rate (share of cases whose chosen key changes under any reorder), +accuracy in each order, and one row per case with every order asked and the key chosen in it. """ from __future__ import annotations import json +import random import statistics from pathlib import Path from typing import Any @@ -18,6 +24,7 @@ FIT_THRESHOLD = 0.8 MIN_CASES = 8 +MAX_SHUFFLES = 50 def verdict(cases: int, accuracy: float) -> str: @@ -115,3 +122,107 @@ def render(summary: dict[str, Any]) -> str: f"verdict: {summary['verdict']}" ) return "\n".join(lines) + + +def parse_order(text: str) -> tuple[str, int]: + """``reverse``, ``shuffle:N`` or ``same:N`` (1 <= N <= ``MAX_SHUFFLES``) as ``(kind, count)``; else a ValueError. + + ``same:N`` is the control: N more asks in the file order, so a "flip" there is the backend disagreeing with + itself (sampling, a served model behind a load balancer, GPU non-determinism), not position bias. + """ + if text == "reverse": + return "reverse", 1 + kind, sep, count = text.partition(":") + if kind in ("shuffle", "same") and sep and count.isdigit() and 1 <= int(count) <= MAX_SHUFFLES: + return kind, int(count) + raise ValueError(f"--order must be reverse, shuffle:N or same:N with 1 <= N <= {MAX_SHUFFLES}, got {text!r}") + + +def reorderings(options: dict[str, str], kind: str, count: int, rng: random.Random) -> list[dict[str, str]]: + """The reordered copies of ``options`` to ask: same keys and texts, only the dict order differs. + + ``reverse`` gives one copy, ``same`` gives ``count`` copies in the file order (the control). ``shuffle`` gives ``count`` seeded permutations, each different from the file order + when the options allow it (a one-option case has no other order, so its copy equals the original). + """ + keys = list(options) + if kind == "reverse": + return [{key: options[key] for key in reversed(keys)}] + if kind == "same": # the non-determinism control: the file order again, ``count`` times + return [dict(options) for _ in range(count)] + out: list[dict[str, str]] = [] + for _ in range(count): + order = keys[:] + for _ in range(20): # resample the identity; with two or more keys this ends almost surely + rng.shuffle(order) + if order != keys or len(keys) < 2: + break + out.append({key: options[key] for key in order}) + return out + + +async def run_order( + cases: list[dict[str, Any]], decision_model: DecisionModel, order: str, *, seed: int = 0 +) -> dict[str, Any]: + """Each case in file order, then in every reordering ``order`` asks for; flip rate, per-order accuracy, rows.""" + kind, count = parse_order(order) + rng = random.Random(seed) + rows: list[dict[str, Any]] = [] + for case in cases: + original = await pick(decision_model, state=case["state"], options=case["options"], rules=case["rules"]) + asks = [{"options": list(case["options"]), "choice": original["choice"], "ms": original["ms"]}] + for options in reorderings(case["options"], kind, count, rng): + answer = await pick(decision_model, state=case["state"], options=options, rules=case["rules"]) + asks.append({"options": list(options), "choice": answer["choice"], "ms": answer["ms"]}) + reordered = asks[1:] + rows.append( + { + "note": str(case.get("note") or ""), + "choice": original["choice"], + "ok": original["choice"] in case["accept"], + "reordered_choices": [ask["choice"] for ask in reordered], + "reordered_ok": sum(ask["choice"] in case["accept"] for ask in reordered), + "flipped": any(ask["choice"] != original["choice"] for ask in reordered), + "asks": asks, + } + ) + cases_asked = len(rows) + reordered_asks = sum(len(row["reordered_choices"]) for row in rows) + right = sum(row["ok"] for row in rows) + accuracy = right / cases_asked if cases_asked else 0.0 + flips = sum(row["flipped"] for row in rows) + all_ms = [ask["ms"] for row in rows for ask in row["asks"]] + return { + "order": order, + "seed": seed, + "rows": rows, + "cases": cases_asked, + "flips": flips, + "flip_rate": round(flips / cases_asked, 3) if cases_asked else 0.0, + "right": right, + "accuracy": round(accuracy, 3), + "reordered_asks": reordered_asks, + "reordered_right": sum(row["reordered_ok"] for row in rows), + "reordered_accuracy": round(sum(row["reordered_ok"] for row in rows) / reordered_asks, 3) + if reordered_asks + else 0.0, + "median_ms": int(statistics.median(all_ms)) if all_ms else 0, + "verdict": verdict(cases_asked, accuracy), # from the file order, as without --order + } + + +def render_order(summary: dict[str, Any]) -> str: + """The order check as text: one line per case (file-order answer, reordered answers, FLIP), then the totals.""" + lines = [f"{'case':<28}{'answer':<16}{'':<6}{'reordered':<24}{'':<5}"] + for row in summary["rows"]: + reordered = ",".join(row["reordered_choices"]) + lines.append( + f"{row['note'][:27]:<28}{row['choice'][:15]:<16}{'ok' if row['ok'] else 'WRONG':<6}" + f"{reordered[:23]:<24}{'FLIP' if row['flipped'] else '':<5}" + ) + lines.append( + f"order: {summary['order']} (seed {summary['seed']}) flips: {summary['flips']}/{summary['cases']} " + f"({summary['flip_rate']:.0%}) accuracy file order: {summary['accuracy']:.0%} " + f"reordered: {summary['reordered_right']}/{summary['reordered_asks']} ({summary['reordered_accuracy']:.0%}) " + f"median ms: {summary['median_ms']} verdict: {summary['verdict']}" + ) + return "\n".join(lines) diff --git a/tests/test_probe_order.py b/tests/test_probe_order.py new file mode 100644 index 0000000..6bbbbe5 --- /dev/null +++ b/tests/test_probe_order.py @@ -0,0 +1,204 @@ +# coding: utf-8 +"""The fit probe's option-order check (#90): flip rate under reversed or shuffled options, keys and texts unchanged.""" + +from __future__ import annotations + +import asyncio +import contextlib +import io +import json +import os +import random +import tempfile +from pathlib import Path +from typing import Any +from unittest import TestCase +from unittest.mock import patch + +from s1a import cli, probe +from s1a.decision_models import JevModel, ScriptedTransport + + +def _cases(n: int = 8) -> list[dict[str, Any]]: + return [ + { + "state": {"i": i}, + "options": {"a": "first text", "b": "second text", "c": "third text"}, + "rules": "pick b", + "accept": ["b"], + "note": f"case {i}", + } + for i in range(n) + ] + + +class _AlternatingTransport(ScriptedTransport): + """Answers ``a`` then ``b`` then ``a``...: a stand-in for a non-deterministic backend.""" + + def __init__(self) -> None: + super().__init__() + self._calls = 0 + + def _answer(self, question: Any) -> Any: + self._calls += 1 + self._choose = "a" if self._calls % 2 else "b" + return super()._answer(question) + + +def _run(transport: ScriptedTransport, order: str, cases: list[dict[str, Any]] | None = None) -> dict[str, Any]: + return asyncio.run(probe.run_order(cases or _cases(), JevModel(transport), order, seed=3)) + + +class TestParseOrder(TestCase): + def test_reverse_and_shuffle_parse_and_anything_else_is_a_value_error(self) -> None: + self.assertEqual(probe.parse_order("reverse"), ("reverse", 1)) + self.assertEqual(probe.parse_order("shuffle:4"), ("shuffle", 4)) + self.assertEqual(probe.parse_order("same:2"), ("same", 2)) + for bad in ( + "shuffle", + "shuffle:0", + "same", + "same:0", + f"same:{probe.MAX_SHUFFLES + 1}", + "shuffle:x", + f"shuffle:{probe.MAX_SHUFFLES + 1}", + "sort", + "", + ): + with self.subTest(bad=bad), self.assertRaises(ValueError): + probe.parse_order(bad) + + +class TestReorderings(TestCase): + def test_keys_and_texts_are_identical_only_the_order_changes(self) -> None: + options = {"x": "ex", "y": "why", "z": "zed", "w": "double u"} + (reversed_,) = probe.reorderings(options, "reverse", 1, random.Random(0)) + self.assertEqual(list(reversed_), ["w", "z", "y", "x"]) + self.assertEqual(reversed_, options) # dict equality ignores order: same keys, same texts + shuffled = probe.reorderings(options, "shuffle", 10, random.Random(0)) + self.assertEqual(len(shuffled), 10) + for copy in shuffled: + self.assertEqual(copy, options) + self.assertNotEqual(list(copy), list(options)) + + def test_same_repeats_the_file_order(self) -> None: + options = {"x": "ex", "y": "why", "z": "zed"} + copies = probe.reorderings(options, "same", 3, random.Random(0)) + self.assertEqual([list(copy) for copy in copies], [["x", "y", "z"]] * 3) + self.assertEqual(copies, [options] * 3) + + def test_a_one_option_case_has_no_other_order(self) -> None: + self.assertEqual(probe.reorderings({"only": "t"}, "shuffle", 2, random.Random(0)), [{"only": "t"}] * 2) + + +class TestRunOrder(TestCase): + def test_a_position_biased_backend_flips_every_case_under_reverse(self) -> None: + transport = ScriptedTransport() # always the first offered key + summary = _run(transport, "reverse") + self.assertEqual((summary["flips"], summary["cases"], summary["flip_rate"]), (8, 8, 1.0)) + self.assertEqual({row["choice"] for row in summary["rows"]}, {"a"}) + self.assertEqual({tuple(row["reordered_choices"]) for row in summary["rows"]}, {("c",)}) + self.assertEqual((summary["accuracy"], summary["reordered_accuracy"]), (0.0, 0.0)) + # the request bodies show both orders with byte-identical keys and texts + first, second = (body["questions"]["pick"]["criteria"] for body in transport.bodies[:2]) + self.assertEqual((list(first), list(second)), (["a", "b", "c"], ["c", "b", "a"])) + self.assertEqual(first, second) + self.assertEqual(transport.bodies[0]["state"], transport.bodies[1]["state"]) + self.assertEqual(summary["rows"][0]["asks"][1]["options"], ["c", "b", "a"]) + + def test_a_content_based_backend_never_flips(self) -> None: + for order in ("reverse", "shuffle:5"): + with self.subTest(order=order): + summary = _run(ScriptedTransport(choose="b"), order) + self.assertEqual((summary["flips"], summary["flip_rate"]), (0, 0.0)) + self.assertEqual((summary["accuracy"], summary["reordered_accuracy"]), (1.0, 1.0)) + self.assertEqual(summary["verdict"], "fits") + + def test_same_is_the_control_position_bias_alone_never_flips_but_noise_does(self) -> None: + biased = _run(ScriptedTransport(), "same:2") # always the first key: biased, but deterministic + self.assertEqual((biased["flips"], biased["reordered_asks"]), (0, 16)) + # a backend that disagrees with itself on every other call flips even though the order never changed + noisy = _run(_AlternatingTransport(), "same:1") + self.assertEqual((noisy["flips"], noisy["flip_rate"]), (8, 1.0)) + self.assertEqual({tuple(row["asks"][1]["options"]) for row in noisy["rows"]}, {("a", "b", "c")}) + + def test_shuffle_asks_n_reorders_per_case_and_is_seeded(self) -> None: + a, b = _run(ScriptedTransport(), "shuffle:3"), _run(ScriptedTransport(), "shuffle:3") + self.assertEqual(a["reordered_asks"], 24) + self.assertEqual([row["asks"] for row in a["rows"]], [row["asks"] for row in b["rows"]]) + self.assertEqual(a["flip_rate"], 1.0) # every shuffle differs from the file order, so the first key moves + self.assertIn("FLIP", probe.render_order(a)) + self.assertIn("flips: 8/8 (100%)", probe.render_order(a)) + + +def _main(argv: list[str]) -> tuple[int, str, str]: + out, err = io.StringIO(), io.StringIO() + with patch.dict(os.environ), contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = cli.main(argv) + return code, out.getvalue(), err.getvalue() + + +class TestProbeCli(TestCase): + def _cases_file(self, tmp: str) -> Path: + path = Path(tmp) / "cases.jsonl" + path.write_text("".join(json.dumps(case) + "\n" for case in _cases()), encoding="utf-8") + return path + + def test_without_order_the_probe_output_and_exit_code_are_unchanged(self) -> None: + with ( + tempfile.TemporaryDirectory() as tmp, + patch.object(cli, "build_model", lambda *_a, **_k: JevModel(ScriptedTransport(choose="b"))), + ): + code, out, _err = _main(["probe", str(self._cases_file(tmp))]) + self.assertEqual(code, 0) + self.assertNotIn("flips", out) + self.assertIn("verdict: fits", out) + with ( + tempfile.TemporaryDirectory() as tmp, + patch.object(cli, "build_model", lambda *_a, **_k: JevModel(ScriptedTransport())), + ): + self.assertEqual(_main(["probe", str(self._cases_file(tmp))])[0], 1) + + def test_order_prints_the_flip_rate_and_writes_every_order_to_json(self) -> None: + with ( + tempfile.TemporaryDirectory() as tmp, + patch.object(cli, "build_model", lambda *_a, **_k: JevModel(ScriptedTransport())), + ): + out_path = Path(tmp) / "order.json" + code, out, _err = _main( + ["probe", str(self._cases_file(tmp)), "--order", "reverse", "--json", str(out_path)] + ) + saved = json.loads(out_path.read_text(encoding="utf-8")) + self.assertEqual(code, 1) # verdict from the file order: the first key is always wrong here + self.assertIn("flips: 8/8 (100%)", out) + self.assertEqual((saved["model"], saved["order"], saved["flip_rate"]), ("jev", "reverse", 1.0)) + self.assertEqual(saved["rows"][0]["asks"][0]["options"], ["a", "b", "c"]) + + def test_a_bad_order_is_a_usage_error_before_any_model_loads(self) -> None: + built: list[str] = [] + + def build(name: str, **_kwargs: Any) -> JevModel: + built.append(name) + return JevModel(ScriptedTransport()) + + with tempfile.TemporaryDirectory() as tmp, patch.object(cli, "build_model", build): + code, out, err = _main(["probe", str(self._cases_file(tmp)), "--order", "sideways"]) + self.assertEqual((code, out, built), (2, "", [])) + self.assertEqual(len(err.strip().splitlines()), 1) + + def test_seed_or_json_without_order_is_a_usage_error_before_any_model_loads(self) -> None: + built: list[str] = [] + + def build(name: str, **_kwargs: Any) -> JevModel: + built.append(name) + return JevModel(ScriptedTransport()) + + for extra in (["--seed", "1"], ["--json", "out.json"]): + with ( + self.subTest(extra=extra), + tempfile.TemporaryDirectory() as tmp, + patch.object(cli, "build_model", build), + ): + code, out, err = _main(["probe", str(self._cases_file(tmp)), *extra]) + self.assertEqual((code, out, built), (2, "", [])) + self.assertIn("only apply with --order", err)