Repository navigation
[Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059
Description
Problem: within one server process, repeating an identical temperature-0 request gives different next-token log-probabilities on a ~100-token prompt. This still happens with every switch DETAILS.md lists for byte-identical repeats (#410). On a near-tie the top token changes from one read to the next. A 26-token prompt repeats exactly.
Environment:
- Strata 0.1.49.2, engine built from [Other] serve: logprobs on /v1/completions (stacks #3332) #4056 (
main7dd5cf1 + [Engine] OpenAI logprobs / top_logprobs / allowed_token_ids for the CUDA/HIP engine #3200 + [Other] add legacy /v1/completions for raw-prompt clients (#2277) #3332). The engine C++ ismain+ [Engine] OpenAI logprobs / top_logprobs / allowed_token_ids for the CUDA/HIP engine #3200, which only adds the LP line printout. - Linux, RTX 5080 16 GB (sm_120), CUDA 13.4, driver 615.78, 123 GB RAM.
- Qwen3.8-Flash-Next IQ3_S. The engine args are those
strata launchwrites:--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp … --max-context 262144 --kv int8 --ple-io ram --kv-resident 32768, no--vision, and--parallel 1. - Also with
--prompt-cache 0 --adapt-swaps 0 --pcie-frac 0and"env": {"STRATA_IQ_MT_MIN": "1"}in the config.
Reproduce: start the server, then send the same request 10 times in a row, with nothing in between:
POST /v1/completions
{"prompt": [<the 101 token ids below>], "max_tokens": 1, "temperature": 0,
"logprobs": 0, "allowed_token_ids": [16, 17, 18, 19], "return_tokens_as_token_ids": true}The prompt is AnyJev's rendering of a 4-level score question ("How urgent is this?", state "My card was charged twice.") through the Qwen3.8 chat template with enable_thinking=False. It is 101 tokens and ends in <think>\n\n</think>\n\n:
[248045, 8678, 198, 2523, 513, 264, 5307, 709, 13, 1394, 668, 381, 2574, 264, 1528, 321, 799, 3296, 13, 17308, 440, 279, 4087, 2306, 1132, 25, 874, 4105, 11, 874, 59429, 11, 874, 15673, 13, 248046, 198, 248045, 846, 198, 1349, 25, 198, 4888, 3575, 557, 11102, 10598, 13, 271, 14162, 25, 2500, 32556, 369, 411, 30, 198, 35728, 279, 2119, 421, 16298, 318, 1719, 5684, 513, 11128, 1590, 198, 16, 13, 524, 198, 17, 13, 3238, 198, 18, 13, 10728, 198, 19, 13, 1496, 198, 15666, 440, 279, 1324, 1132, 13, 248046, 198, 248045, 74455, 198, 248068, 271, 248069, 271]
logprobs: 20 without allowed_token_ids varies in the same way (two reads: label 16 −4.0393 then −4.0616), so it is not the named-id readout.
Expected: with the #410 switches on, the same request repeated within one process gives the same logprobs, as it does for a short prompt.
Actual: with all the switches on, over 10 identical reads:
| prompt | spread (max − min) per label | top token |
|---|---|---|
| 26 tokens ("Write one word: a colour of the sky on a sunny day.") | 0, 0 | always the same |
| 101 tokens (above), labels 16/17/18/19 | 0.53, 0.29, 0.11, 0.13 | 18 or 19 (−0.75 vs −0.75) |
Three reads of the 101-token prompt, shown as the four labels' logprobs (the third also had logprobs: 20):
[-3.97, -3.81, -0.750, -0.750], [-3.72, -3.70, -0.865, -0.673], [-4.07, -3.97, -0.839, -0.664].
With the launch defaults (no switches), the 26-token prompt also changes once a different prompt with the same prefix ran in between (Blue −0.0662 → −0.0698). That part is what #410 describes.
Not isolated yet: --spec 0 / no --mtp, --expert-cache 0, a fixed --prefill chunk, and at what prompt length it starts (26 is exact and 101 varies with the switches on; with the defaults, 73-token prompts varied too). I can run any of these on this PC if that helps.
Why it matters: decision clients (AnyJev's StrataBackend, nokia-applied-research/AnyJev#20) read a typed choice from this one distribution, so a near-tie can resolve differently on identical input.
Reproduction status: reproduced locally, repeatedly.
Activity
gdevenyi commentedon Oct 25, 2026
Reproduced on a second machine with a different GPU generation and build. On this card it also shows up on a short prompt.
Environment:
e0eb04e6with CUDA 12.9,-DSTRATA_EXPERIMENTAL_SM60=ON,-DCMAKE_CUDA_ARCHITECTURES=61, host compilerg++-14.--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp … --kv int8 --ple-io ram --kv-resident 32768, no--vision,--parallel 1.STRATA_IQ_MT_MIN=1, 0.1.32) #410 switches were not set on this run.Reproduce: the 28-token prompt, sent unchanged 8 times in a row, then the same prompt as token ids 8 times:
Actual: the top token is
Bevery time, but its logprob changes every few requests:The values come in runs of 2 or 4 identical reads, then move. All requests are the same 28 tokens (
usage.prompt_tokens28). Reads taken back to back agree to 6 decimals across/v1/chat/completions, a text prompt and a token-id prompt. So the step depends on time or request count, not on the route. With the same launch defaults on epic (RTX 5080, 16 GB), this 28-token prompt repeated exactly.Also on epic (launch defaults, re-run at
e0eb04e6): on ~100-token prompts, the first read of a new prompt differs from the reads after it. Repeating the read, through either path, gave the same value every time. 4 of 50 labels moved by more than 0.1, by up to 0.23, all of them unlikely labels (logprob −3.4 to −5.7). That fits the prompt-cache resume that #410 describes.What this adds to the report:
Not run yet: the #410 switches (
--prompt-cache 0 --adapt-swaps 0 --pcie-frac 0,STRATA_IQ_MT_MIN=1) on monster. I can run that on this machine if it helps separate the adaptive tier from whatever remains on epic.Reproduction status: reproduced locally, on two machines.
leonardomurillo commentedon Oct 25, 2026
I could reproduce this on three other cards, and the switch that removes it is the prefill CPU share:
STRATA_PREFILL_CPU_SHARE=0, or a fixed value such as0.5. It isn't in the #410 list.Setup
--expert-cache auto --prefill auto --spec 4 --mtp … --kv int8 --kv-resident 32768 --max-context 262144, plus your switches:STRATA_IQ_MT_MIN=1 --prompt-cache 0 --adapt-swaps 0 --pcie-frac 0.GEN 1 logprobs=20, one-shot), 10 times in a row with nothing between. Every read was a full read (0 tokens reused).Result
Spread (max − min) of the four labels' logprobs over the 10 reads:
STRATA_PREFILL_CPU_SHARE=0STRATA_PREFILL_CPU_SHARE=0.53and4.--mtpis not involved: without it (--spec 2, as a native pack needs) the spreads were the same, and adding the share switch made them exact.Why, as far as I can read the code
Since 0.1.41 the share is on by default for CUDA builds with one GPU and no batch slots, for chunks below 1024 tokens. It runs as
auto: each layer's split of the streamed experts between the CPU pool and the GPU follows running means of measured CPU and GPU time (cpu_share_env/cpu_share_maxinsrc/prefill/prefill.cpp, unchanged on today'smain). So which experts the CPU computes, and how they round, changes from one read to the next. A fixed share takes the timing out of the decision.This may also fit your follow-up: with the launch defaults, reads after the first resume from the prompt cache and skip the prefill where the share runs, while
--prompt-cache 0makes every read a full prefill. I didn't measure the resume case here. On the 1080 Ti, addingSTRATA_PREFILL_CPU_SHARE=0next to the #410 switches should separate the share from the adaptive tier.Cost
Median time of a one-token read, in ms, at 101 / 965 prompt tokens (one run each, so roughly ±5 %):
Turning the share off costs 15-40 % on these prompts. A fixed 0.5 kept the default's speed here.
Suggestion
Add
STRATA_PREFILL_CPU_SHARE=0to the byte-identical list in DETAILS.md. If speed matters, mention that a fixed value also repeats. One caveat: an explicit value also lifts the share's chunk limit to 3072 (cpu_share_max), and I didn't measure prompts over 965 tokens.Not run: sm_120, an engine newer than 0.1.43, Linux,
/v1/completions.ibbobud commentedon Oct 27, 2026
Following up from #4056 and #3200 — I'm interested in helping with reproducibility testing on older NVIDIA hardware.
I saw @leonardomurillo's findings around STRATA_PREFILL_CPU_SHARE, particularly that a fixed value of 0.5 produced consistent logprobs without the performance penalty of disabling CPU sharing entirely. That's an interesting result.
I'm running dual Titan Xp GPUs (Pascal, sm_61) with Qwen3.8-Flash-Next, and I'd be happy to reproduce these tests on my hardware.
I can compare automatic versus fixed CPU prefill sharing, repeated identical requests, and different prompt lengths, including latency measurements.
I haven't run these tests yet, but I'd be glad to help validate whether the findings hold on Pascal and with the current #4056 branch.
If there's a preferred test matrix or particular configuration you'd like validated, let me know. Happy to contribute my time and results.
leonardomurillo commentedon Oct 27, 2026
@ibbobud thanks, Pascal results would help. They also cover the part gdevenyi's 1080 Ti run left open. Here is the matrix I'd run, in the same shape as the runs above, so the results line up:
Request: gdevenyi's
/v1/completionsbody:max_tokens: 1,temperature: 0,logprobs: 1(ortop_logprobs: 5on chat), the same prompt sent 8 times in a row. Record the top logprob to 6 decimals and the prompt time (usageor wall clock).Prompts: about 28 tokens (the fruit prompt above), about 100, and about 1,000. If you have time, add one of about 3,000: a fixed
STRATA_PREFILL_CPU_SHAREalso raises the chunk limit to 3072, and I have not tested anything above 965 tokens.Cells:
STRATA_PREFILL_CPU_SHARE--adapt-swaps00.500.50Everything else stays at the launch defaults, with
--parallel 1. Restart the server between cells.What each cell tells us:
0cost 15-40 % more prompt time, while0.5cost nothing.Two things worth noting with the results: whether the CPU prefill share is in play on your build at all (if B, C and A give the same prompt times, it probably is not), and how much VRAM is free once the model is loaded.
ibbobud commentedon Oct 27, 2026
@leonardomurillo Below are the results on a single Nvidia Titan Xp on SM_61 pascal.
I had codex run it while I was at work and Astra verify it. Let me know if you need the raw log files or additional data.
Pascal results for #4056
e0eb04e61a45da49610ae33477722c997c077a3b(engine 0.1.49.2): one TITAN Xp 12 GiB, sm_61, Qwen3.8-Flash-Next GSQ-RCO IQ3_S, Xeon W-2123 / 64 GB RAM, Linux driver 580.178.04, CUDA 12.9.1/nvcc 12.9.86, GCC 13.3.0.160/160 requested inference requests succeeded, including the long prompt. Eight sequential reads per prompt/cell, one separate 26-token warmup per process, restarts between cells, order A–E.
/v1/completions,max_tokens: 1,temperature: 0,logprobs: 1,--parallel 1; frozen token-ID prompts andreturn_tokens_as_token_ids: true.Cells are your requested CPU share / adaptive swaps: A auto/default, B 0/default, C 0.5/default, D auto/0, E 0.5/0.
Selected-token logprob spread (max−min), default caching:
No winner changed. Repeats reused all but seven tokens, so I added a separate full-prefill diagnostic with 160/160 successful requests, setting only
--prompt-cache 0in every cell. Metrics confirmed zero reuse:Only E repeated within each cohort at every length in both suites. In the full-prefill suite, A and D flipped between
3and4on the 101-token urgency prompt. Those ranges span the winning token's values, not one fixed label. The C/E and D/E comparisons support contributions from adaptive swapping and automatic sharing under this recipe; they do not directly trace the internal cause of each change. The engine prints LP lines to six decimals; internal bit identity was not measured. Values can differ across cache policies.Full-prefill median prompt time, ms:
Sharing off versus auto cost 78% at 101 and 35% at 981 tokens. Fixed 0.5 versus auto was +3.3% / −4.5% there, but +5.8% at 2983. One fixed-order cohort; small differences are not a general speed ranking or an isolated comparison with Ada.
Automatic-sharing startup messages were present, with three AVX2 workers; runtime
stream_all_minwas 1024 for A/B/D and 3072 for C/E. Exact CPU expert counts are not exposed. Loaded VRAM: 11,641 MiB used / 543 MiB free; peak 11,855 MiB, with low-headroom warnings retained.Common settings followed the issue's launch recipe: automatic cache/prefill, MTP/spec 4, spec-min-p 0.5, context 262144, int8 KV/resident 32768, no vision/control vector. This 64 GB host required default SSD-backed PLE; an initial explicit PLE-RAM attempt failed before readiness and is excluded/logged. All measured containers used a 58 GiB limit.
No engine changes. This #4056 revision predates the newer #3200 GPU scorer. Harness, frozen prompts, per-request CSV/JSON and logs are prepared to share.
leonardomurillo commentedon Oct 27, 2026
The same check on 0.2.7 (engine 0.1.52), three Ada cards (RTX 4090, 4080, 4080 SUPER). Windows, Qwen3.8-Flash-Next IQ3_XXS,
--adapt-swaps 0, one request at a time.0.5.STRATA_PREFILL_CPU_SHARE=0.5, two sessions on the 4080 gave identical values. Two sessions of our 0.1.43-based build on the 4090 did as well.On these cards
0.5is also faster on longer prompts. Prompt tok/s, whole request, fresh prompt each time:Decode stayed within the run-to-run spread. On 0.2.7 there is one run with
0.5on the 4090 and on the 4080 SUPER, and two on the 4080. The 4090 also has two runs of our 0.1.43-based build with0.5, with the same gain (2,390 and 3,915 tok/s), and its chat replies were byte-identical to the default share's. The three PCs have different CPUs, which probably explains why the gain differs.@ibbobud thanks, that's a clean set, and it lines up with ours. Every Ada run above used
--adapt-swaps 0, so they are your D and E cells, and the0.5runs are E. Your C cell covers what we didn't test: with adaptive swaps left on,0.5alone doesn't make the values repeat. On speed the TITAN Xp doesn't show the Ada gain (0.5was 3 % slower at 101 tokens, 4 % faster at 981 and 6 % slower at 2,983), so the prefill gain looks host-dependent rather than general.gdevenyi commentedon Oct 29, 2026
Ran @leonardomurillo's matrix (A–E) on both of my machines. It covers what was listed as not run: sm_120, Linux, an engine newer than 0.1.43, and
/v1/completions. Monster also covers the 11 GB Pascal card.Setup
d032be99(main46c152a1+ [Other] add legacy /v1/completions for raw-prompt clients (#2277) #3332 + the completions logprobs). Qwen3.8-Flash-Next IQ3_S, launch args asstrata launchwrites them (--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp … --kv int8 --ple-io ram --kv-resident 32768, no--vision),--parallel 1.-DSTRATA_EXPERIMENTAL_SM60=ON, Linux./v1/completions(max_tokens: 1,temperature: 0,logprobs: 1, frozen token ids,return_tokens_as_token_ids).prompt N tokens = R reused + K read in X msline for each read. The second suite adds only--prompt-cache 0, which gave 0 reused tokens on every read.0/default, C0.5/default, D auto/--adapt-swaps 0, E0.5/--adapt-swaps 0.epic (RTX 5080, 16 GB)
Spread (max − min) of the chosen token's logprob over 8 reads, default caching:
Median prompt time (engine log, ms), default caching:
Reused tokens per read (default): 28: 0–21; 101: 0–94; 1032: 0–1025; 3051: 0–3044. ⚑ = the chosen token changed between reads.
Spread (max − min) of the chosen token's logprob over 8 reads,
--prompt-cache 0(full prompt read every time):Median prompt time (engine log, ms),
--prompt-cache 0(full prompt read every time):Reused tokens per read (nocache): 28: 0–0; 101: 0–0; 1032: 0–0; 3051: 0–0. ⚑ = the chosen token changed between reads.
monster (GTX 1080 Ti, 11 GB)
Spread (max − min) of the chosen token's logprob over 8 reads, default caching:
Median prompt time (engine log, ms), default caching:
Reused tokens per read (default): 28: 0–21; 101: 0–94; 1032: 0–1025; 3051: 0–3044. ⚑ = the chosen token changed between reads.
Spread (max − min) of the chosen token's logprob over 8 reads,
--prompt-cache 0(full prompt read every time):Median prompt time (engine log, ms),
--prompt-cache 0(full prompt read every time):Reused tokens per read (nocache): 28: 0–0; 101: 0–0; 1032: 0–0; 3051: 0–0. ⚑ = the chosen token changed between reads.
What this shows
--adapt-swaps 0alone) is already exact everywhere. That fits the Ada result rather than contradicting it: every Ada column already had--adapt-swaps 0in it (the Greedy output varies between server requests, but not between fresh engine runs (STRATA_IQ_MT_MIN=1, 0.1.32) #410 switches), so fixing the share was the last step there.--prompt-cache 0, D still drifts at 101 tokens on both cards (0.145 on epic with the top token flipping, 0.093 on monster). D is exact at 1,032 and 3,051. Those prompts start with a chunk of 1,024 tokens, and the auto share only acts below that, which fits./v1/completionsand a token-id prompt give identical values (epicB−0.027356, monster −0.035613). AnyJev's parity check against the server's own top-20 is0.000000(2 runs, epic). With the launch defaults, the same comparison differed by up to 1.14.Suggestion, same as @leonardomurillo's with one addition: the byte-identical list in DETAILS.md should have
--adapt-swaps 0and a fixedSTRATA_PREFILL_CPU_SHARE(0.5 cost nothing measurable here). Either one alone did not make these cards repeat. For prompts close to 3,072 tokens, a value that leaves the default chunk limit alone might be worth having; I haven't measured that.On #4056 I asked whether a request could get a repeatable read without the whole server giving up these features. Both sources here are server-wide state (expert placement, and a share learned from timings), so I didn't try anything per request. If a per-request switch is ever wanted, those are the two things it would have to pin.
Raw per-read JSON and engine logs are available if useful.
gdevenyi commentedon Oct 29, 2026
Correction and follow-up to my matrix above.
--adapt-swaps 0with a fixedSTRATA_PREFILL_CPU_SHAREis not enough on its own, and it isn't free.1. A third source: the first run after a start. The matrix gave each fresh server a single warm-up request, so it missed this.
--adapt-swaps 0,STRATA_PREFILL_CPU_SHARE=0.5.Decider's rotations. I ran it 3 times in a row against one fresh server, logging the engine'sprompt N tokens = R reused + K readline for each read. Every read was a full read (0 reused), so the prompt cache plays no part.--adapt-swaps 0, share 0.5)--pcie-frac 0So the engine settles during its first few dozen requests after a start, then repeats.
--pcie-frac 0(missed experts on the CPU instead of read over PCIe by the GPU) removes it. This is the "first answer after a start" effect DETAILS.md already describes for #410. The expert-cache hit rates printed after each request were the same in run 1 and run 2 (60.4%, 60.3%, 63.0%, …), so the cached experts weren't what changed.2. Cost: I only measured prompt time before. Decode is where the cost is.
predicted_per_second, epic, one fresh server per row.STRATA_PREFILL_CPU_SHARE=0.5--adapt-swaps 0--adapt-swaps 0+ share 0.5 (cell E)--pcie-frac 0Prompt reads cost nothing extra with
--pcie-frac 0(median 400 vs 395 ms over 195 reads). So the full repeatable recipe on this card is--adapt-swaps 0 --pcie-frac 0 STRATA_PREFILL_CPU_SHARE=0.5. That is free for one-token decision reads and costs about 38% of decode speed for anything that generates text. Almost all of that cost comes from turning off the adaptive tier.What that leaves. All three sources come from the same thing: an expert gives slightly different results depending on whether the CPU or the GPU computes it, and which one does changes with the adaptive tier, the PCIe share and the prefill share. Repeatable reads therefore need either:
I haven't found a setting for the second. If a per-request deterministic mode is ever considered, it would have to pin all three. Not run: monster (the 1080 Ti) with
--pcie-frac 0.gdevenyi commentedon Oct 29, 2026
Follow-up: there is now a per-request fix with no decode cost: #4604, canonical reads (draft).
A logprobs request with
max_tokens: 1, or any request with"strata_deterministic": true, is computed the same way whatever the expert cache holds. Every other request keeps the adaptive tier, the PCIe share, the prefill CPU share and the tail skip, so the ~38% decode cost of the server-wide recipe in my previous comment is gone.Bisecting on hardware found two more sources beyond the three above:
STRATA_ROUTE_TAIL_SKIP. It's on by default for CUDA serve, and it drops low-ranked missed experts, so which experts are skipped depends on the cache. It was on in every matrix cell above. That may be part of why--adapt-swaps 0looked like the main source: freezing the tier also freezes which experts are missed.Results with #4604 at Strata's launch defaults (adaptive tier, tail skip, PCIe and CPU shares, prompt cache all on), Qwen3.8-Flash-Next IQ3_S:
-DSTRATA_EXPERIMENTAL_SM60=ON).Canonical values are repeatable for one build, GPU and settings, not across GPUs. Making cached prefixes reusable by canonical reads (checkpoints at fixed aligned positions) is the follow-up I listed in #4604.