[Feature]: Cache Cua-S1's leading prompt block across requests #188

The feature, motivation and pitch

Cua-S1's text worker builds each question's prompt as the system prompt, the question's goal and a short header, then the accessibility tree. An agent asks the same questions at every step, so that leading block (174–700 tokens) is computed again on every request. This is the cross-request case in #150's section 4, and #185's traces now show how much of the work it is.

I'd like to add an opt-in cache that keeps the prefix state at the end of that block across requests. It would live in the shared Qwen executor and be used first by the Cua-S1 text worker, with exports whose LoRA adapter is merged into the base weights (cua-s1-text-merged/1, "merged-adapter" below).

Measured on #185's traces (System1-Agents episodes rendered in Cua-S1's text layout), with one checkpoint per prompt and episodes run one at a time:

The times are means over 52 of these prompts on one RTX 6000 Ada, with a Cua-S1 4B 0.2 merged-adapter text export at main 9466e0e, both run eagerly with the fixed GEMMs. A hit was timed as forward_shared with the block as prefix and the rest twice, minus the same with the rest once. Every hit equalled forward_fixed on the whole prompt bit for bit.

What I have in mind:

• The worker names the checkpoint: the 64-token floor of where the accessibility tree starts. No search over every boundary.
• An entry is the state forward_shared already captures for a prefix: about 50 MiB plus 32 KiB per token on Qwen3.5-4B, about 70 MiB for a 640-token block. 512 MiB held every useful checkpoint with episodes run one at a time; with 8 episodes interleaved, Flights needed about 2 GiB and the forms 4 GiB.
• Entries are keyed by their token IDs and evicted least recently used within a byte budget.
• Hits continue with the fixed GEMMs, so with the cache on, results differ from today's default path by rounding, as with DECIDER_PREFIX.
• Off by default: a budget such as CUA_S1_PREFIX_CACHE_BYTES turns it on. Merged-adapter exports only, refused with CUA_S1_GRAPH=1 at first.

@Levius-Fubuki, the recommended profile keeps the adapter unmerged and rejects prefix continuation by design (#143): it matches the reference by running each prompt as one pass. I'd start with merged-adapter exports. If a mode that keeps the FP32 adapter unmerged but allows reuse is worth having, the cache would carry over to it. Does that order work for you?

Alternatives

• Sharing within a request only. A request's questions have different goals, so they share little more than the system prompt: 0–10% on these traces.
• A checkpoint every 64 tokens, as block-level prefix caches do. That reaches 32–77%, but held 5–443 GiB on these traces. Every 256 tokens is hit or miss: 77% on blackjack, 0% on the ticket router.
• State-first layouts such as Open-Jev don't need this. open_jev: run shared prefixes once per request #179 already shares within the request there, and a cache would add at most 1.7 points.

Additional context

• The default path with CUDA Graphs took 164.0, 154.7, 80.4, 37.6, 41.8 and 36.3 ms on the same prompts; the eager fixed path was faster on these lengths.
• Open-Jev-9B made the decisions in these traces, so the trajectories follow its choices.
• Cua-S1's screenshot layout puts the image before the goal, so only the 70-token system prompt is ahead of the state there; it isn't covered.
• Related: [RFC]: Next-phase roadmap: prefill-only execution, reuse and scheduling #150, benchmarks: record, measure and replay decision traces #185, jev_vl: add native inference and prefix caching #96 (JEV-VL's cache), cua-s1: reuse screenshot prefixes within a request #184.

I'll follow up with a PR for merged-adapter exports.

Before submitting a new issue...

• Make sure you already searched for relevant issues in the issue tracker, and looked for the answer in the documentation.