[gdevenyi]

Problem and result

A typed decision (Jev/AnyJev style) reads its answer from one token's logprobs and compares them with its other reads: a label prior, a calibrated threshold, the same question in another option order. It therefore needs the same numbers for the same prompt. Today they move from request to request. #4059 measured this on Ada, Pascal and Blackwell cards: up to ~0.5 nat, and the top answer flips on near-ties. The cause is that each expert is computed by one of several paths, and the paths round differently:

• the GPU hit path (resident) or PCIe (missed), depending on what the expert cache holds, which the adaptive tier and the prompt path's slot loan keep changing;
• the CPU, depending on the PCIe share and the prompt's CPU share;
• or not at all, for a low-ranked missed expert (STRATA_ROUTE_TAIL_SKIP).

Until now the only fix was server-wide: --adapt-swaps 0 --pcie-frac 0 STRATA_PREFILL_CPU_SHARE=…. That costs every request about 35% of decode speed.

This PR makes it per request. A canonical read takes the same path for every expert, whatever the cache holds. Every other request keeps all the adaptive paths, so chat is as fast as before.

 POST /v1/chat/completions | /v1/completions
+  logprobs + max_tokens 1            → canonical read (automatic)
+  "strata_deterministic": true|false → force it on / off for any request
 GEN … logprobs=K allowed=…
+    canon=1                           # an older engine skips the key
 engine, for that request only:
+  prompt: one batched part, every expert on the GPU (prefill CPU share paused, no prompt windows, no turn split)
+  window: every routed expert treated as a miss → over PCIe where pinned, else the CPU pool (fixed per expert)
+          tail skip paused; residency tables cleared for the window and restored before the next adaptive round
+  no cached prefix reused (live session, checkpoints, batch slots, parked conversations)

Related discussion

• Clients: New: StrataBackend (draft, needs Strata logprobs on /v1/completions) nokia-applied-research/AnyJev#20 (StrataBackend, which reads this route with allowed_token_ids and return_tokens_as_token_ids) and New: anyjev.serve --backend strata, the gateway's Decider mode (stacks on #20) nokia-applied-research/AnyJev#21 (its HTTP decision gateway, anyjev.serve --backend strata, one gateway per Strata server). The scope discussion on the AnyJev side is Strata backend (anyjev/backends/strata.py) and a Decider mode for the gateway: scope check (#20, #21) nokia-applied-research/AnyJev#22. The gateway defaults to a content_free label prior, so with canonical reads the same decision request gets bit-identical probabilities however much other traffic came before it. Measured behind a LiteLLM pass-through: the same request twice gave billing 0.9990941558694965 both times on the RTX 5080.
• [Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059: the measurements behind this (cells A–E on three card generations, the first-run effect, the decode cost of the server-wide recipe).
• [Engine] typed decisions: score a schema's token paths instead of generating them #3743: the typed-decision endpoint under discussion. Canonical reads are what such an endpoint needs underneath, and this PR is independent of it.
• Independent of [Other] add legacy /v1/completions for raw-prompt clients (#2277) #3332 / [Other] serve: logprobs on /v1/completions (stacks #3332) #4056. It works on /v1/chat/completions as is, and on /v1/completions once those land (tested combined; see below).
• Scope has not been agreed with a maintainer yet. It's opened as a draft: HIP, SYCL and Metal aren't built, which CONTRIBUTING requires before review.

Approach

How each source was found (bisected on hardware, one setting at a time; details in #4059):

1. Tail skip: with it on, the reads differ. It's paused per request (pause_tail_skip).
2. Prefill CPU share: with it on auto, the reads differ. It's paused per request (Prefill::pause_cpu_share). A paused chunk doesn't take the CPU pool, so every expert runs on the GPU, exactly as with STRATA_PREFILL_CPU_SHARE=0, and the share's measured state is left alone.
3. Prompt windows: a window of several prompt tokens can miss more experts than its PCIe staging holds (16), and the overflow falls to the CPU depending on residency. Canonical reads therefore read no prompt part through the windows and take no turn checkpoint (so there's one batched part).
4. Residency in the one-token window:
   • On an RTX 5080 (4,360 slots) the VRAM/PCIe split of these prompts' experts was stable, and steps 1–3 were enough.
   • On a GTX 1080 Ti (1,995 slots; the prompt path borrows 1,747 of them) reads were repeatable from a fresh start, then drifted repeatably as the tier moved experts. With --adapt-swaps 0 they were exact, so residency was the last variable.
   • The window now treats every expert as a miss: host_res and d_res are cleared, the window runs, and both are restored. The pending swaps land first (apply_pending(true)), so nothing is in flight.
   • Each expert then takes the same path every time: PCIe if its blob is pinned, the CPU pool if not (a fixed property).
5. Reuse: a read that resumes from a cached prefix gives other (repeatable) values than a fresh one. Canonical reads reuse nothing.

Compatibility: requests that don't qualify are unchanged, apart from two relaxed atomic loads (the pause flags) on paths they already take. The server always sends canon=1 for qualifying requests; an older engine skips unknown keys.

Validation

Revision: 576badfa on main 46c152a1. Hardware runs used the same C++ plus #3332 + #4056 (for /v1/completions), built from source:

• epic: RTX 5080 16 GB (sm_120), CUDA 13.4, Linux.
• monster: GTX 1080 Ti 11 GB (sm_61), CUDA 12.9, -DSTRATA_EXPERIMENTAL_SM60=ON, g++-14, Linux.

Both ran Qwen3.8-Flash-Next IQ3_S with the launch defaults strata launch writes: --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp … --kv int8 --ple-io ram --kv-resident 32768, --parallel 1, with the adaptive tier, tail skip, PCIe share, CPU share and prompt cache all on.

The workload was AnyJev's parity script, traced: 65 reads per run (15 named-id reads, 15 top-20 reads, the Decider's 35 rotation reads), compared value by value across runs.

Passed:

• epic: 4 runs identical to the bit. Between runs, the cache was churned (a 3,051- and a 1,032-token plain read, 4 chat generations), and the 4th run came after a restart. The Decider's results were identical too.

• monster: the same 4-run protocol, all identical.

• The fix is necessary: before step 4, monster differed on 56 of 65 reads after one run, and on all 65 after chat.

• Chat decode (4 prompts × 256 tokens, median), measured with requests that don't qualify:

  machine	without this PR	with this PR
  epic, same source	117.8 / 121.4 tok/s	118.0–120.3 tok/s
  monster	40.8 tok/s (released 0.1.52)	41.2–41.5 tok/s

• Read cost: a canonical read costs what a first read of its prompt does.

  	canonical	plain, first read	plain, reusing the cache
  epic, 73–101 tokens (engine prompt time)	363 ms	441 ms	–
  monster, 101 tokens (wall time)	1.9 s	1.5 s	0.24 s
  monster, 1,032 tokens (wall time)	4.36 s	4.36 s	0.23 s

  The epic plain read is slower because a plain read splits at the turn and reads the tail through windows.

• python -m unittest discover -s serve -p 'test_*.py': 1,221 tests, 1 failure, the pre-existing GpuChoice.test_hip_ordinal that unmodified main also fails here.

• serve.test_logprobs: OK. That includes 4 new tests for canonical_read / canon=1: auto-detection, non-qualifying requests, the field overriding in both directions, and a non-bool field falling back to auto.

Failed:

• None observed beyond the pre-existing test_hip_ordinal.

Not run:

• HIP, SYCL and Metal builds. The C++ touches expert_source.cpp, prefill.cpp and generate.cpp, which they compile.
• A layer split, peer or remote expert GPUs, and --parallel batch slots with canonical reads. The window override covers the main table and every stage's copy (res_upload), but peer/remote experts aren't misses to the plan, and batch slots weren't tested.
• Windows.

Limits

• No reuse. Repeated reads of one prompt, and decisions sharing a long prefix (the same state, several questions or option orders), pay a full read each time. The follow-up I have in mind is canonical checkpoints at fixed aligned positions, where a canonical read also splits its prompt, so resuming does the same arithmetic as reading fresh. This fits [Engine] typed decisions: score a schema's token paths instead of generating them #3743's "read the prompt once" design.
• Repeatable for the same build, GPU and settings. Not across GPUs or builds: different kernels round differently.
• The window's PCIe traffic: every routed expert of the one window (about 480 blobs × ~2.7 MB for this model) goes over the link: ~25 ms at 58 GB/s, ~110 ms at 12 GB/s. An unpinned expert runs on the CPU, the same way every time.
• Longer replies: with "strata_deterministic": true and max_tokens > 1, the window override applies to every window of the reply, so decode is slower for that request. The automatic detection never does this.

Merge danger

Door: two-way. It's per request and opt-out-able ("strata_deterministic": false), and other requests are unchanged.

Blast radius: small. The one behaviour change for existing clients: a one-token logprobs request now reads its whole prompt (no reuse) and gets canonical values, so its numbers shift slightly once, then stay put.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Cix1XJhNhNNuKXi26KvXnU