[twu3202]

Required human contributor answers

1. What outcome are you trying to achieve?

   Measure how much computation repeats in real agent interactions, and decide which kind of reuse to build.

2. What is the highest-leverage variable for that outcome?

   The share of prefill tokens that each kind of reuse could skip, which depends on how often the same state is asked about again, how long the prompt prefix stays the same between steps, how soon the same screenshot comes back, and how many questions share one state.

3. How will this work move that variable rather than create busywork?

   Each kind of reuse works differently and costs memory and complexity. Measuring first shows which one would cut the most prefill work and latency, so we build that one and skip the ones that would not pay off.

4. Who is the customer or user, and what do they actually need?

   Users whose decision requests repeat computation in different ways, such as agents asking several questions about one page or screenshot, guardrails applying one rubric to many messages, or voice agents asking again as a transcript grows. They need faster answers.

After answering the four questions: is a PR needed at all to achieve this outcome?

Yes. The traces now cover browser, form and tool fronts and two prompt layouts, and their results answer #150's open question, so the tools and results are worth sharing.

For a bug fix: link actual observed failure evidence.

N/A, not a bug fix.

Purpose

For #150's first section: find the repeated work in real agent interactions. Its first step is a trace format and a replay script under benchmarks/; this adds those, a recorder for System1-Agents, a metrics script, two small traces and the first results.

• tracefile.py: the s1-trace/1 format. One record per attempt, with episode, step, retries, send and receive times, the think time since the previous response, the deadline, the response and image hashes (no image bytes, no token ids). id, group and request match the manifest rows proposed in [Feat] Replay changed-image decision requests with retained evidence ThinkFlowLab/system1-agents#72.
• record_s1a.py: runs with System1-Agents' Python and wraps its HTTP decision client and tool loop from outside its source.
• metrics.py: renders each request as the native Open-Jev or Cua-S1 worker does and tokenizes it with the worker's export, leaving out requests the worker would refuse. It reports what sharing within a request (the rule of Decider's prefix.rs) and a cache of earlier prompts (64-token chunks, per episode or as the server sees them) would skip, and repeated states and images.
• replay.py: one client per episode, in recorded order with its think time, episodes concurrently or one after another; per-request latency, deadline misses and prefix_cache, with bench.py's evidence files and percentiles. It is a separate script because test: add weekly local GPU accuracy and latency checks #141 and Add a Public231 fast-screen before full C1 #181 are changing bench.py.
• samples/: 20 blackjack hands and one ticket-router run. results-2026-10-25.md: the first results and how they were recorded.

For #150's open question, branch batching or the cross-request cache first: in these traces a cache pays only where each question's goal comes before the state (Cua-S1). Where the state comes first (Open-Jev), sharing within a request is nearly all of the reuse.

Test Plan

System1-Omni Version / Commit: c18e9e7 on main 05075d0.

CPU: the CI steps (format, Clippy, frontend tests, workspace tests, release build, Python checks) and mkdocs build --strict. The new tests cover the format checks, the planner against prefix.rs's rules, the cache rule, refused and failed requests, both workers' prompts (Open-Jev against tests/open_jev/data/contract.json, Cua-S1 through its contract.py), the recorder against a stand-in System1-Agents package, paced replay against a local HTTP server (order, think time, deadlines, offsets, retries), and the samples. With the 9B export's tokenizer, the Open-Jev token ids also match tests/open_jev/data/tokenization.json.

Recordings, on one RTX 6000 Ada: System1-Agents 9b20b98 fronts (Google Flights live, its local form fixtures, 2048, millionaire, blackjack, ticket router), with Open-Jev-9B deciding (native worker with #179's change, no prefix switch) and Qwen3-8B planning through vLLM 0.32 with thinking off. Each trace was measured in both layouts.

Test Result

CPU. All pass on macOS.

Shares of prefill tokens that could be skipped, each layout on its own tokens:

Trace	Episodes / measured decisions	Open-Jev: within a request	+ server cache, last 8	Cua-S1: within a request	+ server cache, last 8
Google Flights, live	28 / 93	89.2%	90.8%	4.4%	31.8%
zurich-london (System1-Agents' Jev 1.13 recording)	1 / 10	91.5%	91.9%	4.7%	35.6%
Form fixtures	36 / 108	60.5%	61.6%	9.5%	49.4%
2048	5 / 800	58.2%	59.7%	0%	46.6%
Ticket router	1 / 30	59.8%	59.8%	0%	45.3%
Blackjack	100 / 119	41.8%	41.8%	0%	77.2%

No state came back exactly. Five Flights results pages were over the 9B export's 4,096-token limit and are not measured. The Cua-S1 worker runs no planner, so its "within a request" column is hypothetical. Blackjack's 77% needs a cache that ignores episode boundaries: each hand is its own short episode (84 of 100 take one decision). Agents sent one request at a time; without a chat model, the next request followed the last response within milliseconds.

Replaying the Flights and form traces against #179 is in that PR's comment.

Trajectories depend on the deciding model: Open-Jev-9B ends 22 of the 28 Flights episodes at the second decision.

Demo / evidence

The Flights traces copy third-party pages, and the form and 2048 traces carry prompt text adapted from browser-use (MIT), so they stay outside the repository; the results file says how they were recorded. The two samples are System1-Agents' content (Apache-2.0), and the ticket-router row can be recomputed from its sample.

Self-review

• I have reviewed the full diff and addressed the issues I found.
• I have checked that the change follows the project's architecture and stays focused on the stated purpose.
• I have run the checks appropriate to this change and reported commands, results, and anything I could not verify above.
• I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.