diff --git a/benchmarks/README.md b/benchmarks/README.md index bccf2ec..c0f83c8 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -30,9 +30,13 @@ python -m benchmarks.legs.leg1_public231 aggregate \ --out /absolute/path/to/results/open-jev-9b ``` -Omni native Open-Jev uses `/v1/systemone`; Intern's official service uses -`/v1/decisions`. `--model` is optional. The collector preserves state, questions and -option order, sends `thinking.enabled=false`, and never sends targets or provenance. +The public231 collector uses the typed `state` / `questions` / `thinking` request +and reads `answers.decision`. Omni native text Open-Jev exposes this contract at +`/v1/systemone`; Intern's official service uses `/v1/decisions`. Omni's request +and response schema depends on the selected worker: native JEV-27B-VL instead +uses the flat contract described below at the same `/v1/systemone` path. +`--model` is optional. The collector preserves state, questions and option order, +sends `thinking.enabled=false`, and never sends targets or provenance. It makes one request at a time, without retries. It does not start services or choose GPUs. `environment.json` describes the **actual service**, without credentials: @@ -75,6 +79,72 @@ consistency before re-scoring. Orphan or missing raw files are errors. It never an existing summary; use a new output directory for another recomputation. These checks detect inconsistent evidence, not coordinated rewriting of an unsigned archive. +## Changed-image replay + +Replay a frozen screenshot sequence using the flat `kind` / `state` / `question` / +`options` contract of official JEV `/v1/decide` or Omni native JEV-27B-VL +`/v1/systemone`, using the same `DecisionClient` without a model override. This +collector does not send the public231 typed contract to Intern `/v1/decisions`: + +```bash +python -m benchmarks.legs.changed_image_replay \ + --manifest /absolute/path/to/sequence.jsonl \ + --endpoint http://127.0.0.1:8000 --path /v1/systemone \ + --run-meta /absolute/path/to/environment.json \ + --cache-state unknown --out /absolute/path/to/results/image-replay +``` + +Each JSONL line has a unique nonempty `id`, a nonempty `group`, and the original +`request` object. For example (replace the placeholder with actual base64 bytes): + +```json +{"id":"frame-1","group":"game-over","request":{"kind":"choice","state":["Inspect the screenshot.",{"image":"data:image/png;base64,..."}],"question":"Game over?","options":["No","Yes"]}} +``` + +`state` is a list with at least one inline PNG/JPEG image, either `{"image": URI}` +or `{"type":"image_url","image_url":{"url": URI}}`. Preflight checks every row, +base64 encoding and MIME signature before any request; image decoding stays with +the service. Remote URLs are rejected. `choice` requires 2–256 string options; +`noul` and `score` omit options. The original request, image order and option order +are preserved. Repeated requests with distinct IDs are sent again. There are no +warmups, retries, deduplication, service restarts or cache resets. + +Run metadata requires `checkpoint.repo/revision` and `backend.name/version` from +the actual service. Include source revisions, hardware and cache configuration +as available. `--cache-state` records only the caller's declaration: the caller +must arrange a cold service, and resetting counters does not empty its caches. +Keep any canonical-ID alias map alongside the external input evidence. + +The fresh output directory must be outside this repository. It retains the exact +`manifest.jsonl`, `environment.json`, `run.meta.json`, one `raw/NNNNNN.json` per +attempt and `results.jsonl`. Result records include sequence/id/group/kind, +`request_sha256` (of the request serialized by `benchmarks.evidence.dump`), ordered +`image_uri_sha256`, `response_body_sha256`, and the raw file's path and hash. +The full request, image URIs and original response text are in raw evidence, +together with HTTP status, decoded response, client error, timestamp and latency. +`completion.json` counts planned/attempted/remaining/succeeded/failed and records +the stop reason. +HTTP 400/422 refusals and malformed answers remain failed records; access errors, +other non-2xx responses, transport failures and interruptions stop the sequence. + +Response validation checks kind/effective_kind, ordered options, probability +count/range, a probability sum within `1e-6` of 1, and choice/index consistency. +The sum tolerance only allows floating-point rounding, including float32 output; +it is not a reference-comparison tolerance. Unnormalized answers remain +`parse_error` records with their raw evidence and count as failures. `noul` uses +`false,true`; `score` uses `0` through `5`. `ok` and `succeeded` mean only a valid +response; they do not establish reference agreement, cache correctness or task +completion, and do not add three-kind support to the Agents `DecisionModel` API. +Use the independent Omni harness for numerical reference comparison. Comparing +repeated images requires the complete request to match, including all state text, +image order, question, options and inference settings. Compare the cache path +with the reference or cache-disabled path under the same checkpoint and runtime; +require deterministic inference settings or account for expected nondeterminism +before attributing differences to caching. Matching image URI hashes alone is +insufficient. Client `latency_s` ends after `response.text` is read, before JSON +decoding; it excludes service startup. Short replays do not establish stable p95 +latency, and replaying 2048 screenshots does not establish autonomous game completion. + ## Tests ```bash diff --git a/benchmarks/legs/changed_image_replay.py b/benchmarks/legs/changed_image_replay.py new file mode 100644 index 0000000..8889f2b --- /dev/null +++ b/benchmarks/legs/changed_image_replay.py @@ -0,0 +1,224 @@ +"""Replay a frozen inline-image JSONL sequence and retain every HTTP attempt.""" + +import argparse +import base64 +from contextlib import closing +import json +import math +import os +from pathlib import Path +import time + +from benchmarks.client import DecisionClient, decode +from benchmarks.evidence import digest, dump, outside_repo, write_json + + +def image_hashes(request): + state = request.get("state") + if not isinstance(state, list): + raise ValueError("state must be a list containing inline PNG/JPEG images") + hashes = [] + for part in state: + if not isinstance(part, dict): + continue + if "image" in part: + uri = part["image"] + elif part.get("type") == "image_url": + image = part.get("image_url") + uri = image.get("url") if isinstance(image, dict) else None + else: + continue + if not isinstance(uri, str): + raise ValueError("image URI must be a string") + header, separator, payload = uri.partition(",") + signatures = {"data:image/png;base64": b"\x89PNG\r\n\x1a\n", "data:image/jpeg;base64": b"\xff\xd8\xff"} + if not separator or header not in signatures: + raise ValueError("Only inline base64 PNG/JPEG images are allowed") + data = base64.b64decode(payload, validate=True) + if not data.startswith(signatures[header]): + raise ValueError("Image bytes do not match their PNG/JPEG MIME signature") + hashes.append(digest(uri.encode("utf-8"))) + if not hashes: + raise ValueError("Each request must contain at least one inline image") + return hashes + + +def load_manifest(source): + rows, seen = [], set() + for number, line in enumerate(source.splitlines(), 1): + try: + row = decode(line) + if not isinstance(row, dict) or any( + not isinstance(row.get(key), str) or not row[key] for key in ("id", "group") + ): + raise ValueError("Each row requires nonempty id and group strings") + if row["id"] in seen: + raise ValueError("Duplicate input id") + request = row.get("request") + if not isinstance(request, dict) or request.get("kind") not in ("choice", "noul", "score"): + raise ValueError("request.kind must be choice, noul or score") + if not isinstance(request.get("question"), str): + raise ValueError("request.question must be a string") + if request["kind"] == "choice": + options = request.get("options") + if ( + not isinstance(options, list) + or not 2 <= len(options) <= 256 + or any(not isinstance(option, str) for option in options) + ): + raise ValueError("choice requires 2–256 string options") + elif "options" in request: + raise ValueError("options is only allowed for choice") + hashes = image_hashes(request) + dump(row).encode("utf-8") + except ValueError as exc: + raise ValueError(f"Manifest line {number}: {exc}") from exc + seen.add(row["id"]) + rows.append((row, hashes)) + if not rows: + raise ValueError("Manifest is empty") + return rows + + +def response_error(request, response): + if not isinstance(response, dict): + return "Expected a JSON response object" + kind = request["kind"] + options = request.get("options", ["false", "true"] if kind == "noul" else list("012345")) + if response.get("kind") != kind or response.get("effective_kind") != kind: + return "Missing or mismatched kind/effective_kind" + if response.get("options") != options: + return "Response options differ from the requested kind or option order" + probabilities = response.get("probabilities") + if ( + not isinstance(probabilities, list) + or len(probabilities) != len(options) + or any(isinstance(p, bool) or not isinstance(p, (int, float)) or not 0 <= p <= 1 for p in probabilities) + ): + return "Expected one finite probability in [0, 1] per option" + if not math.isclose(math.fsum(probabilities), 1.0, rel_tol=0, abs_tol=1e-6): + return "Expected probabilities to sum to 1" + index = response.get("choice_index") + if isinstance(index, bool) or not isinstance(index, int) or not 0 <= index < len(options): + return "choice_index is outside the options" + if response.get("choice") != options[index]: + return "choice differs from options[choice_index]" + return None + + +def record_for(row, hashes, sequence, raw, raw_path, output): + code = raw["http_status"] + if raw["client_error"] is not None: + status, error = "client_error", raw["client_error"] + elif code is None or not 200 <= code < 300: + status, error = ("refused" if code in (400, 422) else "http_error"), raw["response_body"] + else: + error = response_error(row["request"], raw["response"]) + status = "parse_error" if error else "ok" + text = raw["response_body"] + return dict( + sequence=sequence, + id=row["id"], + group=row["group"], + kind=row["request"]["kind"], + request_sha256=digest(dump(raw["request"]).encode("utf-8")), + image_uri_sha256=hashes, + response_body_sha256=digest(text.encode("utf-8")) if text is not None else None, + raw_file=str(raw_path.relative_to(output)), + raw_sha256=digest(raw_path.read_bytes()), + ts=raw["ts"], + latency_s=raw["latency_s"], + status_code=code, + status=status, + ok=status == "ok", + error=error, + ) + + +def predict(args): + source = args.manifest.read_bytes() + rows = load_manifest(source) + environment = args.run_meta.read_bytes() + meta = decode(environment) + for group, fields in (("checkpoint", ("repo", "revision")), ("backend", ("name", "version"))): + values = meta.get(group) if isinstance(meta, dict) else None + if not isinstance(values, dict) or any(not isinstance(values.get(k), str) or not values[k] for k in fields): + raise ValueError(f"run-meta requires actual {group}: {fields}") + if args.path not in ("/v1/decide", "/v1/systemone"): + raise ValueError("Choose /v1/decide or /v1/systemone explicitly") + if args.cache_state not in ("unknown", "cold", "warm"): + raise ValueError("cache-state must be unknown, cold or warm") + output = outside_repo(args.out) + with closing(DecisionClient(args.endpoint, path=args.path, model=None, timeout_s=args.timeout)) as client: + output.mkdir(parents=True, exist_ok=False) + (output / "raw").mkdir() + (output / "manifest.jsonl").write_bytes(source) + (output / "environment.json").write_bytes(environment) + write_json( + output / "run.meta.json", + dict( + endpoint=args.endpoint, + path=args.path, + environment_sha256=digest(environment), + manifest_sha256=digest(source), + planned=len(rows), + timeout_s=args.timeout, + cache_state_declared=args.cache_state, + order="manifest sequential; no warmup, retries or deduplication", + latency_basis="client HTTP POST through response.text; excludes JSON decode and service startup", + ts=time.time(), + ), + ) + attempted, succeeded, stop = 0, 0, None + try: + with (output / "results.jsonl").open("x", encoding="utf-8") as stream: + for sequence, (row, hashes) in enumerate(rows, 1): + attempted += 1 + raw = client.request(row["request"]) + raw_path = output / "raw" / f"{sequence:06d}.json" + # Escaping preserves decoded lone surrogates from malformed service responses. + with raw_path.open("x", encoding="utf-8") as raw_stream: + raw_stream.write(json.dumps(raw, ensure_ascii=True, allow_nan=False) + "\n") + record = record_for(row, hashes, sequence, raw, raw_path, output) + stream.write(dump(record) + "\n") + stream.flush() + os.fsync(stream.fileno()) + succeeded += record["ok"] + if record["status"] in ("client_error", "http_error"): + stop = raw["client_error"] or f"HTTP {raw['http_status']}" + break + except KeyboardInterrupt: + stop = "interrupted" + except Exception as exc: + stop = f"{type(exc).__name__}: {exc}" + raise + finally: + write_json( + output / "completion.json", + dict( + planned=len(rows), + attempted=attempted, + remaining=len(rows) - attempted, + succeeded=succeeded, + failed=attempted - succeeded, + stop_reason=stop, + ts=time.time(), + ), + ) + return output + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--manifest", type=Path, required=True) + parser.add_argument("--endpoint", required=True) + parser.add_argument("--path", required=True, choices=("/v1/decide", "/v1/systemone")) + parser.add_argument("--run-meta", type=Path, required=True) + parser.add_argument("--out", type=Path, required=True) + parser.add_argument("--timeout", type=float, default=120) + parser.add_argument("--cache-state", choices=("unknown", "cold", "warm"), default="unknown") + print(predict(parser.parse_args())) + + +if __name__ == "__main__": + main() diff --git a/tests/benchmarks/test_changed_image_replay.py b/tests/benchmarks/test_changed_image_replay.py new file mode 100644 index 0000000..1b1dd74 --- /dev/null +++ b/tests/benchmarks/test_changed_image_replay.py @@ -0,0 +1,300 @@ +"""Offline replay contract checks; no model inference or external HTTP traffic.""" + +import json +from types import SimpleNamespace + +import httpx +import pytest + +from benchmarks.client import DecisionClient +from benchmarks.evidence import digest, dump +from benchmarks.legs import changed_image_replay as replay + + +PNG = ( + "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Wl6nH0AAAAASUVORK5CYII=" +) +JPEG = "data:image/jpeg;base64,/9j/2Q==" + + +def item(identity="frame-1", kind="choice", image=PNG): + request = {"kind": kind, "state": ["Inspect this frame.", {"image": image}], "question": "Game over?"} + if kind == "choice": + request["options"] = ["No", "Yes"] + return {"id": identity, "group": "screenshots", "request": request} + + +def answer(request): + kind = request["kind"] + options = request.get("options", ["false", "true"] if kind == "noul" else list("012345")) + return { + "kind": kind, + "effective_kind": kind, + "options": options, + "probabilities": [1.0] + [0.0] * (len(options) - 1), + "choice_index": 0, + "choice": options[0], + "model": "CPU fixture, no inference", + } + + +@pytest.fixture +def setup(tmp_path, monkeypatch): + calls = [] + rows = [item()] + manifest = tmp_path / "input.jsonl" + environment = tmp_path / "actual-environment.json" + environment.write_text( + dump({"checkpoint": {"repo": "fixture", "revision": "test"}, "backend": {"name": "mock", "version": "1"}}) + ) + args = SimpleNamespace( + manifest=manifest, + run_meta=environment, + endpoint="http://fixture.invalid", + path="/v1/decide", + timeout=1, + cache_state="unknown", + out=tmp_path / "output", + ) + + def configure(new_rows=None, handler=None): + if new_rows is not None: + rows[:] = new_rows + manifest.write_text("".join(dump(row) + "\n" for row in rows), encoding="utf-8") + + def respond(request): + calls.append(request) + return handler(request) if handler else httpx.Response(200, json=answer(json.loads(request.content))) + + class FixtureClient(DecisionClient): + def __init__(self, *positional, **kwargs): + assert kwargs.get("model") is None + super().__init__(*positional, **kwargs, transport=httpx.MockTransport(respond)) + + monkeypatch.setattr(replay, "DecisionClient", FixtureClient) + return args, calls + + return configure + + +def evidence(output): + rows = [json.loads(line) for line in (output / "results.jsonl").read_text().splitlines()] + raw = [json.loads((output / row["raw_file"]).read_text()) for row in rows] + completion = json.loads((output / "completion.json").read_text()) + return rows, raw, completion + + +@pytest.mark.parametrize("path", ["/v1/decide", "/v1/systemone"]) +def test_serial_three_kinds_preserve_repeated_images_options_and_evidence(setup, path): + inputs = [item("first"), item("second", "noul", JPEG), item("third", "score"), item("reversed"), item("again")] + inputs[3]["request"]["options"].reverse() + inputs[2]["request"]["state"][1] = {"type": "image_url", "image_url": {"url": PNG}} + args, calls = setup(inputs) + args.path, args.cache_state = path, "cold" + output = replay.predict(args) + records, raw, complete = evidence(output) + assert [json.loads(call.content) for call in calls] == [row["request"] for row in inputs] + assert all(str(call.url) == args.endpoint + path for call in calls) + assert [row["sequence"] for row in records] == [1, 2, 3, 4, 5] + assert [row["id"] for row in records] == [row["id"] for row in inputs] + assert [row["kind"] for row in records] == ["choice", "noul", "score", "choice", "choice"] + assert all(row["ok"] and row["status"] == "ok" for row in records) + assert all(row["latency_s"] > 0 for row in records) + for row, original, response in zip(records, inputs, raw): + assert row["request_sha256"] == digest(dump(original["request"]).encode()) + assert row["raw_sha256"] == digest((output / row["raw_file"]).read_bytes()) + assert row["response_body_sha256"] == digest(response["response_body"].encode()) + assert response["request"] == original["request"] + assert json.loads(response["response_body"]) == response["response"] + assert records[0]["image_uri_sha256"] == records[2]["image_uri_sha256"] == records[3]["image_uri_sha256"] + assert (output / "manifest.jsonl").read_bytes() == args.manifest.read_bytes() + assert (output / "environment.json").read_bytes() == args.run_meta.read_bytes() + meta = json.loads((output / "run.meta.json").read_text()) + assert not {"argv", "environment", "environment_file", "manifest_file"} & meta.keys() + assert meta["cache_state_declared"] == "cold" + assert meta["manifest_sha256"] == digest(args.manifest.read_bytes()) + assert meta["environment_sha256"] == digest(args.run_meta.read_bytes()) + assert complete["planned"] == complete["attempted"] == complete["succeeded"] == 5 + assert complete["remaining"] == complete["failed"] == 0 and complete["stop_reason"] is None + + +def test_multiple_images_retain_uri_hash_order(setup): + row = item() + row["request"]["state"].extend([{"image": JPEG}, {"image": PNG}]) + args, calls = setup([row]) + records, _, _ = evidence(replay.predict(args)) + assert json.loads(calls[0].content) == row["request"] + assert records[0]["image_uri_sha256"] == [digest(uri.encode()) for uri in (PNG, JPEG, PNG)] + + +@pytest.mark.parametrize( + "broken", + [ + {}, + {"id": "later", "group": "x", "request": []}, + item("frame-1"), + item("later", "other"), + item("later", image="https://example.invalid/image.png"), + item("later", image="data:image/png;base64,not-base64"), + item("later", image="data:image/png;base64,dGV4dA=="), + item("later", image="data:image/gif;base64,R0lGODlh"), + ], +) +def test_bad_later_input_prevents_every_request(setup, broken): + args, calls = setup([item(), broken]) + with pytest.raises(ValueError): + replay.predict(args) + assert not calls and not args.out.exists() + + +@pytest.mark.parametrize("change", ["options", "state", "question", "noul_options"]) +def test_bad_request_fields_preflight(setup, change): + row = item() + if change == "options": + row["request"]["options"] = ["Only one"] + elif change == "state": + row["request"]["state"] = ["No screenshot"] + elif change == "question": + row["request"]["question"] = 3 + else: + row["request"]["kind"] = "noul" + args, calls = setup([row]) + with pytest.raises(ValueError): + replay.predict(args) + assert not calls and not args.out.exists() + + +@pytest.mark.parametrize("source", ["", "not json\n", "{}\n\n"]) +def test_empty_or_invalid_manifest_preflight(setup, source): + args, calls = setup() + args.manifest.write_text(source) + with pytest.raises(ValueError): + replay.predict(args) + assert not calls and not args.out.exists() + + +def test_invalid_unicode_in_later_request_is_rejected_before_http(setup): + args, calls = setup() + row = item("later") + row["request"]["question"] = "\ud800" + args.manifest.write_text(json.dumps(item()) + "\n" + json.dumps(row) + "\n") + with pytest.raises(ValueError): + replay.predict(args) + assert not calls and not args.out.exists() + + +@pytest.mark.parametrize("status,attempted", [(400, 2), (422, 2), (401, 1), (403, 1), (429, 1), (500, 1)]) +def test_http_rejection_and_access_errors_keep_raw_text(setup, status, attempted): + text = ' {"error":{"message":"rejected"}}\n' + args, calls = setup([item(), item("next")], lambda _: httpx.Response(status, text=text)) + output = replay.predict(args) + records, raw, complete = evidence(output) + assert len(calls) == complete["attempted"] == complete["failed"] == attempted + assert complete["remaining"] == 2 - attempted + assert bool(complete["stop_reason"]) is (attempted == 1) + assert all(r["response_body"] == text and r["http_status"] == status for r in raw) + assert all(not r["ok"] and r["error"] for r in records) + + +@pytest.mark.parametrize("error", [httpx.ReadTimeout("fixture timeout"), KeyboardInterrupt()]) +def test_transport_failure_and_interruption_save_completion(setup, error): + def handler(_): + raise error + + args, calls = setup([item(), item("next")], handler) + records, raw, complete = evidence(replay.predict(args)) + assert len(calls) == complete["attempted"] == 1 and complete["remaining"] == 1 + assert raw[0]["client_error"] and raw[0]["response_body"] is None + assert records[0]["status"] == "client_error" and records[0]["latency_s"] > 0 + assert complete["stop_reason"] + + +@pytest.mark.parametrize( + "patch", + [ + {"kind": "noul"}, + {"effective_kind": "noul"}, + {"options": ["Yes", "No"]}, + {"probabilities": [1]}, + {"probabilities": [True, False]}, + {"probabilities": [-1, 2]}, + {"choice_index": True}, + {"choice_index": 2}, + {"choice": "missing"}, + ], +) +def test_malformed_response_structure_is_retained(setup, patch): + body = answer(item()["request"]) | patch + args, _ = setup(handler=lambda _: httpx.Response(200, json=body)) + records, raw, complete = evidence(replay.predict(args)) + assert records[0]["status"] == "parse_error" and not records[0]["ok"] + assert raw[0]["response"] == body and complete["failed"] == 1 + + +@pytest.mark.parametrize("text", ["broken json", '{"probabilities":[NaN]}', "[]"]) +def test_unparseable_response_keeps_original_text(setup, text): + args, _ = setup(handler=lambda _: httpx.Response(200, text=text)) + records, raw, _ = evidence(replay.predict(args)) + assert records[0]["status"] == "parse_error" and raw[0]["response_body"] == text + + +@pytest.mark.parametrize("status", [200, 400, 422]) +def test_escaped_surrogate_response_is_retained_and_sequence_continues(setup, status): + text = '{"error":"\\ud800"}' + + def handler(request): + if len(calls) == 1: + return httpx.Response(status, text=text) + return httpx.Response(200, json=answer(json.loads(request.content))) + + args, calls = setup([item(), item("next")], handler) + records, raw, complete = evidence(replay.predict(args)) + assert len(calls) == complete["attempted"] == 2 and complete["remaining"] == 0 + assert complete["failed"] == complete["succeeded"] == 1 and complete["stop_reason"] is None + assert records[0]["status"] == ("parse_error" if status == 200 else "refused") + assert records[0]["latency_s"] > 0 and not records[0]["ok"] and records[1]["ok"] + assert raw[0]["http_status"] == status and raw[0]["response_body"] == text + assert raw[0]["response"] == {"error": "\ud800"} + assert records[0]["response_body_sha256"] == digest(text.encode()) + assert records[0]["raw_sha256"] == digest((args.out / records[0]["raw_file"]).read_bytes()) + + +@pytest.mark.parametrize("probabilities", [[0.2, 0.8], [0.20000000298023224, 0.800000011920929]]) +def test_structure_check_does_not_compare_reference_values(setup, probabilities): + body = answer(item()["request"]) | {"probabilities": probabilities} + args, _ = setup(handler=lambda _: httpx.Response(200, json=body)) + records, raw, _ = evidence(replay.predict(args)) + assert records[0]["ok"] and raw[0]["response"] == body + + +def test_unnormalized_probabilities_are_failed_and_sequence_continues(setup): + body = answer(item()["request"]) | {"probabilities": [0.2, 0.2]} + + def handler(request): + response = body if len(calls) == 1 else answer(json.loads(request.content)) + return httpx.Response(200, json=response) + + args, calls = setup([item(), item("next")], handler) + records, raw, complete = evidence(replay.predict(args)) + assert len(calls) == complete["attempted"] == 2 and complete["remaining"] == 0 + assert records[0]["status"] == "parse_error" and not records[0]["ok"] + assert "sum to 1" in records[0]["error"] and records[1]["ok"] + assert raw[0]["response"] == body and json.loads(raw[0]["response_body"]) == body + assert complete["failed"] == complete["succeeded"] == 1 and complete["stop_reason"] is None + + +def test_output_never_overwrites_existing_evidence(setup): + args, calls = setup() + replay.predict(args) + before = (args.out / "completion.json").read_bytes() + with pytest.raises(FileExistsError): + replay.predict(args) + assert len(calls) == 1 and (args.out / "completion.json").read_bytes() == before + + +def test_output_must_be_outside_repository(setup): + args, calls = setup() + args.out = replay.Path(replay.__file__).resolve().parents[2] / "rejected-evidence" + with pytest.raises(ValueError, match="outside"): + replay.predict(args) + assert not calls and not args.out.exists()