[xiezhipeng-git]

AI assistance: the implementation, the tests and this description were drafted with an AI coding agent. I checked every claim below against the source, the command output and the logs it came from; every measurement was run on this machine unless the text says otherwise.

Problem and result. /v1/chat/completions on the CUDA/HIP engine accepted logprobs / top_logprobs and returned HTTP 200, but choices[0].logprobs.content was always []. A client that ranks candidate tokens from the model's own distribution — constrained decoding, candidate scoring, tool-choice re-ranking — silently got nothing: the request succeeds, the content is correct, only the field is missing, so the failure is invisible unless the caller asserts on the field. After this change, logprobs: true with top_logprobs 0..20 (the legacy completions form logprobs: N is also accepted) returns one entry per token with its top alternatives, and allowed_token_ids returns a distribution renormalised over exactly the ids the caller named — including the case where the only id named is the one the model chose, which is the one a decision client asks for.

Related discussion. This supersedes #2254, which the 0.2.0-beta tracker reset closed. GitHub will not reopen it (state cannot be changed. The feat/openai-logprobs branch was force-pushed or recreated.), so this PR carries the same change, rebased onto main. #1916 proposed the same feature for CUDA/HIP and is closed. This implementation follows the protocol the SYCL port already speaks (835d30b) and the Metal port (31f2668), so one server-side reader serves all three backends.

It also carries the fix from the review, ad3cff8 on leonardomurillo:logprobs-own-lines, cherry-picked as a4349ec9 with its authorship intact: an LP line belongs to the request that asked for it, so a reply whose tail is held by a stop string cannot read the next request's lines.

What is already built on it. This is no longer only the field itself:

• #4056 adds logprobs to #3332's legacy /v1/completions and states that it stacks on this branch and that one; a decision client reads its label's log-probabilities from that endpoint, so the two halves it needs are in two open PRs.
• #3743 (typed decisions) lists this PR as prior art for the one-token case: allowed_token_ids gives a probability per field, one prompt read per field, and requires the value to be one token. Values longer than one token and several fields per read are what [Engine] typed decisions: score a schema's token paths instead of generating them #3743 adds on top.
• #675 asked for one-token typed decisions via logprobs.

So the surface this PR adds is the one-token baseline the two follow-ups measure against, and the cases below are on the engine they will run on.

Approach.

• Engine half, src/program/generate.cpp: logprobs=K and allowed=a,b,c are two extra keys on the --serve GEN line (an engine that predates them skips them as unknown), and one LP <logprob> [id:logprob ...] line is written after each T <id> in the same flush. The value comes from the verify window's head logits before sampling, penalties and temperature, so it is the model's distribution rather than the sampler's. Off by default: a request without logprobs produces the previous output byte for byte, and the per-token cost is one integer compare.
• Engine half, the named ids (22fe5184): GEN ... allowed=a,b,c parsed the ids into req_allowed and then cleared that vector three lines later, next to req_imgs.clear(), before either decode loop ran. emit_lp_line therefore always took the top-k branch, and allowed_token_ids only ever filtered what was already in top-k — an id outside it was silently dropped. The clear is gone. On Swift 1.5 IQ3_XXS, allowed_token_ids: [16] used to return the chosen id twice at 0.5 each and now returns one entry at logprob 0; ten named digits (0-9, ids 15-24) for the reply 11 used to cover 2 and then 8 of the ten positions and now cover all ten in every entry, sorted, summing to 1.
• Server half, serve/server.py: sampling_keys appends the two keys only when asked; generate reads each token's LP line before yielding the token (a held line is consumed before the next read, so an engine without LP lines loses no lines, and a DONE read while looking for one still ends the request); LogprobEntries decodes token bytes incrementally so the entries joined spell the reply; structured_chunks re-aligns the entries when response_format: json_schema rewrites the content and returns no logprobs rather than a set that does not match.
• Server half, line ownership (a4349ec9): the engine's lp_lines deque is tagged with the request's cancel Event — one per request, handed unchanged from the handler through Service.run to Engine.generate — and LogprobEntries.take() reads only lines carrying its own. A reply's tail, held by the parser or a stop string, is spelled after the request has released the queue; by then the next request may have replaced lp_lines with its own. The tail now reads its own lines and never consumes the next request's.
• Server half, the filter (900a5b82): when allowed_token_ids left no candidate, _renormalise returned None, the entry was appended all the same, and spell() failed on e["token"] — an unstreamed reply came back 400 "the request is malformed", a streamed one stopped after the role chunk. The chosen token is put back when the caller named it (the line carries its logprob on the head and never repeats it among the named ids, which is exactly why naming only the chosen id used to leave nothing to renormalise over); an entry is never None and never dropped, because the reply's tokens have to keep spelling its text; the entry's logprob is the chosen token's own value, not the first named alternative's; and top_logprobs is sorted most likely first, as the unfiltered top-k already is.
• Server half, the batch admission (1878675f): a slot's BT lines carry no LP line, but an admission does — a max_tokens 1 request is asked of the engine as BGEN and read as GEN 1, the single-token path that prints T and its LP line in one flush (generate.cpp:12090). _control() takes that line and keeps it on the request's own lines, held on its cancel Event: the engine's lp_lines is one attribute and several slots read control lines at once, so it cannot be shared between them. A longer reply there is still a 400, for the reason in Limits. The same commit stops refusing logprobs on --replicas: ReplicaEngine.batch totals how many requests its engines run at once and counts a replica without slots as one, so the "does this engine use batch slots" check said yes to a server that has none — every logprobs request got a 400 while its reply was read by the GEN path that does write the lines. batch_slots_on() asks each replica's own figure instead.
• Refusals (b55e9fdc, 26a2d480, all 400 before anything is generated): logprobs on a --parallel / --batch server unless max_tokens is 1 — a slot prints BT <slot> <id> and writes no LP line, so past the admission's first token a reply's entries would cover a prefix only, which a client cannot tell from the model's own answer (1878675f answers the max_tokens 1 case; see below); and an allowed_token_ids id at or past the vocabulary, which emit_lp_line would use to index a logits row out of bounds.
• The cap, taken out (4b7339e1): the 1..4096 bound on max_tokens / max_completion_tokens was ours alone — OpenAI's request schema makes max_tokens optional and ties no limit to logprobs, and vLLM's defaults to 16 or None and its logprobs only check themselves. A reply is already bounded by max_tokens, or by the rest of the context when it is not sent, so the constant was a second bound on top of one that exists, and it answered 400 to requests that would have worked — the OpenAI SDKs send no max_tokens by default. What the constant protected is real, but it is the reply's own length: a reply that is not streamed, or that answers a json_schema (buffered to be validated), holds every entry until it ends, while a streamed one hands each entry out with its chunk and keeps none. A client that wants that memory bounded streams. The previous commit on this branch (eca6513b) had narrowed the bound to the held cases; this one removes it.
• Compatibility: additive and opt-in. A request that does not ask for logprobs gets the same bytes as before, and no default changes. Two client-visible changes, both of them requests that could not have worked: a logprobs request is now answered at any max_tokens — 5000, 0, -1, or the field absent, which is what the OpenAI SDKs send — where before it took a 400 unless it streamed, and logprobs on --parallel is answered for a max_tokens 1 request (the typed decision) where before it was a 400 for every request, with a longer reply there still a 400 rather than a field that comes back missing.
• Docs: one bullet in docs/DETAILS.md under "Using it" (597571c8, updated by eca6513b, 4b7339e1 and 1878675f) — the keys, that a reply is bounded by max_tokens and not by a cap of ours, what allowed_token_ids keeps and renormalises, that a token whose probability could not be read gets no entry rather than a made-up one, and that --parallel answers only a max_tokens 1 request.

Validation.

Branch: 12 commits on e88ee39d, head 1878675f, with two merges of upstream/main (it moved twice today): d4e8211b onto 41cf05a0 (992 files, +133057/−12580) and 7cbff4fd onto 4b9e05a2 (55 files, +422/−133, the 0.2.6 release). They had to happen — GitHub cannot merge a pull request that has conflicts, and the first one did: docs/DETAILS.md had been restructured (−2162/+1069) around the logprobs bullet, which was put back among the server-behaviour bullets of "Using it". The code never conflicted: serve/server.py and src/program/generate.cpp merged on their own both times, and main's changes to them contain no logprob and touch no LP, BT or GEN line format. Against 4b9e05a2 the diff is 5 files, +1099/−27: src/program/generate.cpp +59/−0, serve/server.py +367/−27, serve/test_logprobs.py +541/−0, serve/test_logprobs_parallel.py +117/−0, docs/DETAILS.md +15/−0. src/program/generate.cpp only gains lines: the whole emit_lp_line path is new here, and the req_allowed clear that 22fe5184 takes out was itself introduced by the first commit, so it never shows as a deletion against the base. Nothing else in src/ moves.

No conflict with today's main (7dd5cf18, 24 October): GitHub reports this PR mergeable: true, and the independent run below merged it onto exactly that revision.

Passed:

• Full CUDA build and link, revision 104becfd: cmake --build build-lp --target strata — Release, CMAKE_CUDA_ARCHITECTURES=89, CUDA 13.3 (V13.3.73), MSVC 14.41, Windows 11, RTX 4090 (48 GB, driver 610.88). [154/154] Linking CXX executable strata.exe, exit 0. The engine that answered every hardware run below was rebuilt from this branch after 22fe5184 and is what the server was started with. eca6513b, the merge d4e8211b and the cap's removal touch no C++: that binary is what answered the runs added for them.
• python -m unittest serve.test_logprobs — 32 tests, OK (104becfd had 17; the file now covers the two refusals, the ownership case, unreadable lines, a line whose named ids are all filtered out, naming only the chosen token, a ten-way choice that only the ended reply settles, a held tail of two tokens, any max_tokens on a reply that holds its entries, and no batch slots behind two replicas). On 26a2d480 the ones eca6513b added fail with a 400: a streamed reply with max_tokens of 5000, 0 and none at all. test_logprobs_take_any_max_tokens (200 / 5000 / 0 / −1 / not sent, not streamed) and test_a_json_schema_reply_takes_any_max_tokens (200 / 0 / 5000, streamed and not) replace the two that asserted the 1..4096 400.
• python -m unittest serve.test_logprobs_parallel — 6 tests, OK: a max_tokens 1 decision through a real StrataEngine and Service over HTTP with the fake engine of serve/test_parallel.py behind them (writing an LP line after each T when the GEN / BGEN line asks), then allowed_token_ids on that admission, four decisions at once each getting its own logprob, a request that does not ask, a longer reply answered 400, and an engine that writes no LP lines at all.
• python -m unittest discover -s serve -t . -p "test_*.py" — 1073 tests in 386 s, 1 failure, 1 error, 21 skipped (the merged tree carries main's new tests, which is why the count is no longer 909). Both are known on a clean upstream/main checkout in this environment and touch no logprobs code: serve.test_winjob (running() on a pid this sandbox cannot read) and serve.test_responses.test_server_close_ends_the_sampler (1 != 2, order-sensitive — it passes when its file runs alone, which it did in the 83-test run just before this one). The skips are POSIX-only (signals, /proc), unrelated to this change. The only test files that mention logprobs are the two above.
• Hardware runs, same engine, model Swift-1.5 IQ3_XXS, --spec 4 --spec-min-p 0.5 --kv int8 --kv-resident 32768, temperature: 0, chat_template_kwargs.enable_thinking: false:
  • non-streamed and stream: true, plain request → 6 entries, "".join(token) equal to the content character for character.
  • stop: [".!"], which holds the tail — non-streamed and streamed → content Cafe costs 5., 6 entries each, the held 5. flushed after the request left the queue with its own entries.
  • the review's scenario on the real engine: two overlapping requests, the first one's tail held by its stop string and sent while the second is generating, 4 rounds → 0 mismatched replies, each reply's tokens spelling that reply.
  • allowed_token_ids: [16] (the digit 1, the id the model chose) → 200, one entry, one candidate at 0.0 — previously the 400 "the request was malformed", streamed and not.
  • ten named digits 0-9, reply 11 → 2 entries, 10 candidates each, most likely first, sum(exp(logprob)) = 1. Only the ended reply settles which of 1 and 11 was meant.
  • a three-token label: allowed_token_ids over {0,1}, reply 111 → 3 entries, each over the two named ids, joint path probability 0.999870.
  • 4b7339e1, the reply's length is no longer capped: logprobs: true, top_logprobs: 5, the reply 11, max_tokens absent / 0 / 200 / 4096 / 5000, each of them streamed and not → 200 ten times, 2 entries each, the tokens spelling 11, the non-streamed replies carrying 5 candidates each.
  • 4b7339e1, allowed_token_ids over the ten digits 15-24 with max_tokens: 5000, streamed → 200, 2 entries of 10 candidates spelling 11; allowed_token_ids: [16] (the digit the model chose), not streamed → 200, one entry, one candidate at 0.0 — previously the 400 "the request was malformed".
  • 4b7339e1, json_schema with max_tokens absent / 0 / 200 / 5000, streamed and not → 200 in all eight, 6 or 7 entries spelling {"n":11} / {"n": 11}; before this commit the streamed ones with absent or 5000 were 400.
  • 4b7339e1, allowed_token_ids: [248320] (the vocabulary size) → 400, still.
  • 1878675f, the same machine and engine but --parallel 2: max_tokens: 1 with top_logprobs: 5 → 200, one entry spelling the reply (A −0.062427, then B −2.848, A −7.415); allowed_token_ids over 11 ids → 200, the named ids as the candidates, sum(exp(logprob)) = 1.0, a chosen token outside them keeping its own logprob; max_completion_tokens: 1 → 200 with its entry; stream: true → 200, one chunk carrying the entry for Odd; four decisions at once on two slots → 200 each, 1.6 s for all four, every entry matching its reply; max_tokens 5, max_tokens absent and max_completion_tokens: 0 → 400 naming max_tokens 1; a plain request and a 64-token reply unchanged.
• Independent run, reported by @gdevenyi in this thread (2026-10-25) — not run by me: #3332 merged with this branch onto main 7dd5cf18 (both merged cleanly), CUDA build from setup.py's flags, RTX 5080 (sm_120), CUDA 13.4, Linux, Qwen3.8-Flash-Next IQ3_S, --spec 4 --mtp, --parallel 1. python -m unittest serve.test_logprobs 28/28 OK (the file had 28 tests at 26a2d480); logprobs: true, top_logprobs: 5, max_tokens: 1 returned choices[0].logprobs.content (B −0.018839, then B −4.51, A −5.98) where the 0.1.48 engine returned no logprobs for the same request; allowed_token_ids agreed with the unrestricted top-K and the engine's ids matched the Hugging Face Qwen/Qwen3.8-Flash-Next tokenizer. This is the only run on hardware that covers --mtp: the runs above used --spec 4 --spec-min-p 0.5 without it.

Failed: none observed.

Not run:

• HIP backend build. src/program/generate.cpp is shared by the CUDA and HIP builds; only the CUDA configuration was built here (no AMD GPU or toolchain on this machine). CONTRIBUTING.md asks for every backend a touched file is compiled into — if someone with an AMD card can run the HIP build, that closes the last gap in this section.
• SYCL and Metal builds. Neither is touched: they carry their own implementations (install_logprobs in sycl/serve/server_intel.py), and no Intel or Apple toolchain is present here.
• The BT slot phase of --parallel / --batch: a reply longer than one token there has no entries, and is a 400; see Limits. What is measured is the max_tokens 1 decision that works and the 400 for anything longer.

Limits.

• On --parallel / --batch, only a max_tokens 1 request carries logprobs; a longer reply is a 400, not a reply whose entries stop halfway. A slot prints BT <slot> <id> and its decode loop never calls emit_lp_line, and the line carries no slot id (LP <logprob> [id:logprob ...]), so with N slots interleaved on one stdout an LP line could not be attributed to a request even if one were written: that needs a slot-tagged line in the engine plus the matching read in the server. The admission is the one token that does come with a line — the engine reads BGEN <slot> 1 as GEN 1, the single-token path — which is exactly the typed-decision case this feature exists for. A request that stays solo on such a server does run the GEN path too, but a second request arriving moves the running one into a slot mid-reply (generate_batched), so its entries would stop with no way for the client to tell. This is the same line gt50 drew on OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine #1916: "LP lines for the BT slot phase can be a later step."
• The reported logprob describes the model's distribution before penalties and temperature, not the post-sampling one.
• With response_format: json_schema the returned content may not be spelled by the tokens; in that case no logprobs are returned, by design.
• An LP line that does not parse (nan, -nan(ind), a cut line) or carries a non-finite value gives no entry for that token rather than failing the request: logprobs.content can be shorter than the reply, and a client must pair entries by token, not by index.
• A reply that is not streamed, or that answers a json_schema, holds every entry (its token, bytes and up to 20 alternatives) until it ends, so a long one holds a lot of them in the server's memory. Nothing here caps that: the bound is the reply's own max_tokens, or the rest of the context when it is not sent, as on OpenAI and vLLM, neither of which ties a length limit to logprobs. Streaming is the way to keep that memory off the server's side.
• CUDA/HIP only; the SYCL and Metal implementations are untouched.
• #4056 adds logprobs to /v1/completions on top of this and #3332; #4059 reports run-to-run variation on ~100-token prompts, which is a sampling-determinism matter and is not changed here.

[leonardomurillo]

A reply can be given the next request's logprobs. When a stop string or the parser holds back the end of a reply, that tail is sent after the request has released its turn. If the next request is generating by then, the tail takes the next request's LP lines: the first reply's tokens no longer spell its content, and the second reply loses those entries. Two logprobs requests close together are enough.

Reproduction on 104becf: add Api.test_a_request_reads_only_its_own_lines and its helpers NextEngine and HandOver from the branch below (LpEngine unchanged), then

python -m unittest serve.test_logprobs.Api.test_a_request_reads_only_its_own_lines -v

Request A is "Café costs €5." with stop: [".!"], so its final "." is held; B is "Bonjour!". The hooks fix the order: B pauses after its first token, its LP line in lp_lines, until A's reply is read. Each reply's "".join(token) must equal its content. 4 of 4 runs:

AssertionError: 'Café costs €5B' != 'Café costs €5.'

The path in serve/server.py at 104becf:

• Service._run holds the turn in with turn: (4737, release at 4734); the held tail is sent after it, at 5109 (parser.finish) and 5116 (stops.flush()).
• B's StrataEngine.generate replaces the deque at 2150 and appends its lines at 2218.
• A's with_lp calls lp.spell at 5434; LogprobEntries.take (5361-5362) pops B's lines; on a mismatch spell() returns all pending entries.

The fix:

• lp_lines is an LPLines deque tagged with the request's cancel Event; LogprobEntries gets the same Event and never reads another request's lines.
• logprobs need max_tokens (or max_completion_tokens) 1..4096, else 400, because a non-streamed or json_schema reply holds its entries until it ends.
• a line that does not parse (nan, -nan(ind), a cut line) or is not finite gives no entry; the request goes on.

Branch: https://github.com/leonardomurillo/Strata/tree/logprobs-own-lines (ad3cff8, one commit on 104becf). PR to feat/openai-logprobs: xiezhipeng-git#1

---

Separate note, reproduced with the mock engine, not fixed on the branch: with allowed_token_ids, when no alternative survives the filter, _renormalise returns None, take() appends it, and spell() fails on e["token"]. On 104becf, the test file's LpEngine (alternatives 65, 72, 88, 91) with allowed_token_ids: [66] gives 400 the request is malformed (TypeError: 'NoneType' object is not subscriptable); streamed, the reply stops after the role chunk. Not checked on a real engine.

[xiezhipeng-git]

A reply can be given the next request's logprobs. When a stop string or the parser holds back the end of a reply, that tail is sent after the request has released its turn. If the next request is generating by then, the tail takes the next request's LP lines: the first reply's tokens no longer spell its content, and the second reply loses those entries. Two logprobs requests close together are enough.

Reproduction on 104becf: add Api.test_a_request_reads_only_its_own_lines and its helpers NextEngine and HandOver from the branch below (LpEngine unchanged), then

python -m unittest serve.test_logprobs.Api.test_a_request_reads_only_its_own_lines -v

Request A is "Café costs €5." with stop: [".!"], so its final "." is held; B is "Bonjour!". The hooks fix the order: B pauses after its first token, its LP line in lp_lines, until A's reply is read. Each reply's "".join(token) must equal its content. 4 of 4 runs:

AssertionError: 'Café costs €5B' != 'Café costs €5.'

The path in serve/server.py at 104becf:

• Service._run holds the turn in with turn: (4737, release at 4734); the held tail is sent after it, at 5109 (parser.finish) and 5116 (stops.flush()).
• B's StrataEngine.generate replaces the deque at 2150 and appends its lines at 2218.
• A's with_lp calls lp.spell at 5434; LogprobEntries.take (5361-5362) pops B's lines; on a mismatch spell() returns all pending entries.

The fix:

• lp_lines is an LPLines deque tagged with the request's cancel Event; LogprobEntries gets the same Event and never reads another request's lines.
• logprobs need max_tokens (or max_completion_tokens) 1..4096, else 400, because a non-streamed or json_schema reply holds its entries until it ends.
• a line that does not parse (nan, -nan(ind), a cut line) or is not finite gives no entry; the request goes on.

Branch: https://github.com/leonardomurillo/Strata/tree/logprobs-own-lines (ad3cff8, one commit on 104becf). PR to feat/openai-logprobs: xiezhipeng-git#1

Separate note, reproduced with the mock engine, not fixed on the branch: with allowed_token_ids, when no alternative survives the filter, _renormalise returns None, take() appends it, and spell() fails on e["token"]. On 104becf, the test file's LpEngine (alternatives 65, 72, 88, 91) with allowed_token_ids: [66] gives 400 the request is malformed (TypeError: 'NoneType' object is not subscriptable); streamed, the reply stops after the role chunk. Not checked on a real engine.

Thanks for the review — I finally ran this on real hardware, and it changed two things. Your ad3cff8 is cherry-picked with your authorship kept; everything below is on feat/openai-logprobs (pushed, 26a2d480).

Your held-tail fix holds. With the ownership test in take() replaced by if cur is not self.lines and True: and nothing else changed, Api.test_a_request_reads_only_its_own_lines fails exactly the way you described (B's line, Bonjour! carrying A's held .). With your commit it passes, 6 runs of 6. I then ran the same race on the real engine, ten times over: A asks for a sentence ending in a full stop with stop: [".!"] so the final . is held, B asks for Bonjour!, both at once. 20 replies out of 20 had "".join(token) equal to the content — A came back as The sky is blue., The sun sets., The cat sat on the mat., B always Bonjour!, streamed and not. Your tag is doing what it is meant to do off the mock too.

The one you left open. emit_lp_line puts the chosen token's logprob on the head of the line and names the ids after it, skipping the chosen id, so an empty set is not a mock-only case — but when I ran it, the engine never saw the list at all. GEN ... logprobs=K allowed=a,b,c parsed the ids into req_allowed and then, three lines later and right next to req_imgs.clear(), did req_allowed.clear(), before either decode loop ran. So emit_lp_line always took the top-k branch, and allowed_token_ids only filtered what top-k had already decided to print: ids outside top-k were lost. On this model, naming only id 16 ("1") gave that id twice at -0.6931 each (the chosen token plus its own copy in top-k) instead of one entry at 0; ten named digits (0-9, ids 15-24) with the reply 11 covered 2 and then 8 of them in the two entries (2 and 5 with top_logprobs: 5). That is one line removed in 22fe5184. Measured after it, same model, same requests: [16] gives one entry per token with top_logprobs a one-element list at 0; the ten digits give all ten candidates in every entry, sorted, sum(exp()) = 1.000000, the chosen 1 among them; allowed_token_ids without logprobs is still 200 and never reaches the engine; and "eleven" comes back as the two tokens 1, 1, which spell the content with or without allowed.

With that in place your four _renormalise changes do what they say: the chosen token is put back among the candidates when it was named (probability 1 instead of an empty list), an entry is never None and never dropped (a line carrying none of the named ids keeps the chosen token with its own logprob and top_logprobs: [], so logprobs.content can still spell the reply), e["logprob"] is the chosen token's own value, and top_logprobs is sorted most likely first.

One more from the same path, which came out of reading the code rather than a crash: emit_lp_line indexes its logits row with each named id as given, so an allowed_token_ids entry past the vocabulary would read out of bounds. It is now a 400 before anything is sent (Api.test_allowed_token_ids_outside_the_vocabulary_is_a_400; with a vocabulary of 300, id 300 gets the 400 and id 299 gets 200).

About "vLLM's semantics" in the docstring. I read vLLM's source before deciding: allowed_token_ids becomes logits.masked_fill_(mask, -inf) in apply_logits_processors, before sampling, and the default logprobs_mode="raw_logprobs" reports the pre-mask distribution. So in vLLM the chosen token is always one of the named ids and nothing is renormalised in the report; here the ids filter what is reported and generation is untouched, so the chosen token can be outside them and keeps its own logprob. The docstring says that now, and so does the docs bullet.

--parallel was the other thing I could not answer before, and it needed a new case in the tests. The engine's batch loop prints BT <slot> <id> and never calls emit_lp_line — the only two call sites are the solo loop (14430) and the speculative one (14844) — so a --parallel server could only answer a logprobs request with the field missing, which a client cannot tell apart from a reply that had no tokens. I refused it instead: the same 400 the max_tokens cap raises, before anything is generated. Measured with --parallel 2 on the same card: logprobs streamed and not both 400 with a message naming --parallel; a plain request still 200 with the same content; allowed_token_ids alone still 200 and not sent to the engine. The new test is Api.test_logprobs_are_refused_on_the_batch_path — the first case in the file for a --parallel server, covering a situation neither of us had tested — and it goes red when the refusal is disabled. The docs bullet and the class docstring say it.

Tests and speed. serve/test_logprobs.py is 28 of 28, and python -m unittest discover -s serve -t . -p "test_*.py" in a clean shell is 909 tests, 0 errors, 20 skipped. The two rules your review added are kept and still tested: logprobs without max_tokens/max_completion_tokens in 1..4096 is 400 (missing, 0 and 5000 measured on the card, 4096 accepted), and a line that does not parse — nan, -nan(ind), a cut line — gives no entry and the reply goes on. rebuild-lp.bat (Ninja + MSVC, CUDA) builds and links. With logprobs off, the same 160-token prompt: 135.3 tok/s median here (131.8 / 136.3 / 135.3) against 129.3 on the release engine (127.4-137.1 over six warm runs) — within the spread, so nothing measurable is added when nobody asks. With top_logprobs: 5 it is 112.2 tok/s; the per-token softmax over the vocabulary is real work, and I did not try to hide it.

What I did not run: HIP, SYCL and Metal builds — one CUDA card here, and src/program/generate.cpp is the only engine file touched. I also did not re-measure the old --parallel answer (200 with the field absent) on the card, since the answer there is now the 400; the "no LP lines in the batch loop" half is the two call sites, read.

The docs change is one bullet under "Using it" in docs/DETAILS.md: the keys, where the entries land, that the tokens spell the reply, the 1..4096 cap and its 400, what allowed_token_ids keeps, the out-of-vocabulary 400, and the --parallel refusal. Drop it if you would rather write that section yourself.

[leonardomurillo]

Thank you for running this on real hardware and for keeping the authorship, much appreciated.

The real-engine race (20 of 20 replies spelling their own content) was the part I could not show from the mock, so that closes it for me. Nice catch on req_allowed.clear() too: that explains why the empty-set case never showed up on the engine side. The --parallel refusal is a better answer than a missing field.

I'm closing my PR into your branch, since its commit is already in here. Nothing else from my side on this one.

[gdevenyi]

Tested on hardware: merged with #3332 onto main 7dd5cf1 (both merged cleanly), built for CUDA with setup.py's flags. RTX 5080 (sm_120), CUDA 13.4, Linux, Qwen3.8-Flash-Next IQ3_S, --spec 4 --mtp, --parallel 1.

• python -m unittest serve.test_logprobs: 28/28 OK.
• /v1/chat/completions with logprobs: true, top_logprobs: 5, max_tokens: 1 returns choices[0].logprobs.content: B −0.018839, then B −4.51, A −5.98. Without this PR, the 0.1.48 engine returned no logprobs on the same request.
• allowed_token_ids returned the same values as the unrestricted top-K, and the engine's ids match the Hugging Face Qwen/Qwen3.8-Flash-Next tokenizer.

#4056 builds on this and adds logprobs on #3332's /v1/completions. Separately, run-to-run variation on ~100-token prompts is in #4059. It isn't caused by this PR, but it shows up in these logprobs.

[xiezhipeng-git]

Thanks @gdevenyi — the --mtp half of that is what only your run covers: every hardware run recorded above used --spec 4 --spec-min-p 0.5 without MTP, so a drafting path that proposes several tokens per step is now measured to keep each LP line in step with its T line. Your run is in the Validation section, attributed.

Three things about the field shape, since #4056 has to reproduce them on /v1/completions:

• allowed_token_ids filters what is reported, then renormalises over the named ids, so sum(exp(logprob)) is 1 across that set. vLLM decomposes this differently: allowed_token_ids there is a logits processor that masks generation, and what it reports is a separate switch, --logprobs-mode. Its default raw_logprobs is the values from before any processor, so the mask is not reflected at all; processed_logprobs does reflect it (and those values also sum to 1 over the named set) but brings temperature and top_k/top_p along with it. So neither of its four modes is the point this endpoint reports — mask applied, temperature and penalties not — and at temperature 1 with no top_k/top_p the processed mode lands on the same numbers; the two only diverge once the sampler is reshaped. Your named set agreeing with the unrestricted top-K is the same arithmetic from the other end: the shift is −log(mass of the named set), so a set holding nearly all of the mass shifts by nearly nothing, and a narrower set comes out higher rather than lower.
• A chosen token outside the named ids keeps its own logprob and an empty top_logprobs. The line prints the chosen token's value on the head and never repeats it among the named ids, so the server puts it back when the caller did name it; when the caller did not, there is nothing to renormalise over and the empty top_logprobs is the intended shape, not a missing field.
• Keys per entry: token, logprob, bytes, top_logprobs[] most likely first, and no id. The entries spell the reply, so pair by token rather than by index: an unparseable line leaves that token without an entry, and the list can be shorter than the reply. A logprobs request needs max_tokens / max_completion_tokens in 1..4096, and --parallel refuses logprobs with a 400. If /v1/completions takes logprobs, those two rules are what keep the two endpoints behaving alike.

One scope note so it does not get carried into #4056: --parallel 1 does not take the batch loop, so the --parallel refusal is still unexercised on your hardware. Not a gap in what you ran, just where its edge is.

On the PR itself:

• The description now carries your run, the fact that #3332 plus this branch merge cleanly onto main 7dd5cf18, and a line each for #4056, #3743 and #675.
• #3743 is worth a look before #4056 settles its own field shape: it lists allowed_token_ids here as the one-token typed-decision path, and puts values longer than one token and several fields per prompt read above it. Same caller, same reading of a distribution, two different surfaces.
• No rebase is needed; GitHub reports this mergeable: true against today's main.

One thing that is visible in the checks and is not the code: Title area and Minimal serving runtime (both pull_request) have been action_required since 19 October, while Intake (pull_request_target) runs and succeeds. A fork PR needs those two approved before they execute, so mergeable_state: unstable is two unapproved runs rather than a failing suite.

[gdevenyi]

Thanks, that's useful. #4056 now includes eca6513 (e0eb04e6). Your bound goes through a shared check_logprobs(req, svc, max_req, kept). /v1/completions refuses logprobs with stream: true, so it always keeps the 1..4096 bound, and it uses the same --parallel refusal. serve.test_logprobs: 39/39 (your 30 plus 9 for completions).

On allowed_token_ids: #4056 deliberately does not renormalise on /v1/completions. It reports the named ids' full-vocabulary values, which for those ids are the numbers vLLM's default raw_logprobs gives. A decision client reading /v1/completions expects that, and it keeps "how much mass do my labels hold" readable. Chat keeps your renormalised shape. Agreed that the --parallel refusal is untested on my side: everything ran with --parallel 1.

FYI: since ed4b4db on main this morning, GitHub shows this PR as conflicting in docs/DETAILS.md. #4056 inherits that, and I'll re-merge once you rebase.

[gdevenyi]

Second hardware run, now including eca6513 (tested through #4056 at e0eb04e6). This time on two machines:

	epic	monster
GPU	RTX 5080 16 GB, sm_120	GTX 1080 Ti 11 GB, sm_61 (Pascal)
Build	CUDA 13.4	CUDA 12.9, -DSTRATA_EXPERIMENTAL_SM60=ON, host compiler g++-14
Driver	615.78	580.178

Both ran Qwen3.8-Flash-Next IQ3_S with the args strata launch writes (--spec 4 --mtp … --kv int8 --expert-cache auto), and --parallel 1.

The results were the same on both machines:

• Chat with logprobs and max_tokens: 1, not streamed: entries come back (monster: B −0.0359, B −3.67).
• eca6513: a streamed reply with logprobs and no max_tokens is answered, and its entries spell the reply exactly. With max_tokens: 5000 it is also answered.
• Still bounded: not streamed without max_tokens → 400, and streamed json_schema with max_tokens: 0 → 400.
• serve.test_logprobs: 30/30 of yours (in the 39 with [Other] serve: logprobs on /v1/completions (stacks #3332) #4056's tests). Full serve/ suite: 1,080 tests, 1 failure, GpuChoice.test_hip_ordinal, which also fails on unmodified main here.

So the LP lines stay in step with their T lines on the Pascal / CUDA 12 / SM60 build as well, with MTP on.

One thing for anyone comparing numbers: on monster (11 GB, so more experts move between RAM and VRAM), even a 28-token prompt's logprob drifts every few requests under the default settings. Eight identical requests gave B −0.0363, −0.0363, −0.0432 ×4, −0.0203 ×2. That is the #410 behaviour, written up in #4059, and not this PR. Reads taken back to back agree.

[xiezhipeng-git]

Thanks — Pascal / sm_61 on CUDA 12.9 with --mtp is the widest coverage this has had, and the LP lines staying in step with their T lines there is exactly the part I could not test.

One correction on the version, because two of those lines are eca6513b's behaviour and a later commit changed it.

4b7339e1 takes the bound out entirely, not just for streamed replies. On the head (7cbff4fd):

• not streamed with logprobs: true and no max_tokens → 200 (was 400)
• not streamed with max_tokens: 0 or -1 ("the rest of the context") → 200
• json_schema, streamed or not, with max_tokens: 0 or 5000 → 200

The reason is the one you pointed at earlier: max_tokens already bounds the reply, and without it the reply is bounded by the rest of the context, so 1..4096 was a second bound on top of one that exists. OpenAI has neither (max_tokens optional in the request schema, no link to logprobs) and vLLM has neither (max_tokens defaults to 16 or None, logprobs only checks itself). Measured here on Swift-1.5 IQ3_XXS with --spec 4 --mtp, 22 cases: max_tokens absent / 0 / 200 / 4096 / 5000 × streamed and not — all 200, every token with an entry; json_schema at absent / 0 / 200 / 5000 × both — all 200.

So #4056 at e0eb04e6 still carries a bound this branch has removed, and it is the same hunk:

LOGPROBS_MAX_TOKENS = 4096
def check_logprobs(req, svc, max_req, kept=True):
    if kept and not 0 < max_req <= LOGPROBS_MAX_TOKENS: ...

kept defaulting to True keeps the bound on /v1/completions, and that line will collide with the deletion here when you re-merge. Either way is defensible, but it should be one way and said in the text: if /v1/completions should keep a bound, make it a named choice rather than a default; otherwise drop max_req from the shared check and the constant with it, as the head does.

docs/DETAILS.md is resolved — d4e8211b merged 41cf05a0, 7cbff4fd merged 4b9e05a2 (0.2.6) — so mergeable: true again and re-merging is safe now.

GpuChoice.test_hip_ordinal failing on unmodified main matches what we see; nothing to do. And the 28-token drift on the 11 GB card is a sharper measurement than the ~100-token one in #4059.

[ibbobud]

I'm following this PR along with #3332 and #4056, particularly for using Qwen3.8-Flash-Next for decision scoring.

I'm running Strata on dual Titan Xp GPUs (Pascal, sm_61) and would be happy to help test CUDA logprobs compatibility, performance, and reproducibility on older hardware.

If there are specific test cases that would be useful, I'm glad to contribute my time.

[Niko1221]

Thank you. Fixed in 0.2.7: OpenAI logprobs, top_logprobs and allowed_token_ids on /v1/chat/completions. Your name is on the commit. We could check it with unit tests and builds, but not on your hardware or with your workload. Please try 0.2.7 and tell us how it goes.

Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.7

[xiezhipeng-git]

0.2.7 works on the hardware and workload this was built for: one RTX 4090, Swift-1.5 IQ3_XXS, --spec 4 --spec-min-p 0.5 --mtp --kv int8 --kv-resident 32768 --max-context 131072 --vision, with the 32-bit-tokenizer pack as setup.py writes it.

Released 0.2.7

• /v1/chat/completions returns logprobs and top_logprobs, streamed and non-streamed.
• Across five prompts (short fact, Chinese explanation, code, prose and arithmetic), the entry tokens join exactly to the reply text, with content and reasoning_content checked separately.
• max_tokens is optional: with the field omitted the request still returns HTTP 200 with entries. What it falls back to is the room left in the context, not a fixed default — prepare() takes max_new = room when the request's own value is missing or 0 (serve/server.py:4993).
• allowed_token_ids returns 200 and reports only the named candidates, sorted most likely first. Those values are conditioned, not raw: in our probe the named probabilities sum to 1.0 over the named set.
• One client-facing detail: a one-token reply can be entirely reasoning on this model. max_tokens: 1 on “The capital of France is” returned content: null, reasoning_content: "We"; a client reading only content would mistake that for missing logprobs. With chat_template_kwargs: {"enable_thinking": false} the same request returns content: "The" and one logprobs.content entry, which is the short path for anything that scores a single forced token.
• That two-array mapping is this layer's own choice, and the alternative is worth naming: OpenAI returns no logprobs at all on reasoning models (o-series through the GPT-5 family), and vLLM goes the other way — one flat logprobs.content over every generated token, reasoning included, and, with a reasoning parser configured, logprobs: null once include_reasoning is false. Keying entries by the message field they spell is what keeps logprobs.content joining to message.content, which is the property the five-prompt check above measures. The split is available here because two things line up and both are cheap to join: the serve layer's parser emits a reasoning or a content event for each piece of the reply (serve/server.py:6100-6105), and the engine writes one LP line per token as it emits it, so LogprobEntries.spell() cuts the token run at that piece's character offsets (serve/server.py:5952-5970). The cut is exact when a token boundary falls at both ends of the piece; if a BPE token straddles the boundary (the answer's leading newlines merged with its first word, say), the pending run goes to the earlier piece whole (the fallback at the end of spell), so content's entries can then carry a leading token that is not part of message.content. That did not happen on the five prompts above, and serve/test_logprobs.py's mock tokenizer (one token per byte) cannot produce it, so a straddling token on the real engine is in the "not run" list. vLLM builds its logprobs from the whole token run before its parser splits the text (v0.32.0, vllm/entrypoints/openai/chat_completion/serving.py:643 streamed, :986 not, the parse at :998), so after the split one flat array is left and no per-field one to give; with include_reasoning: false it returns logprobs: null (:702-706 streamed, :1004-1009 not).

@Niko1221
Cost on 0.2.7, and the separate GPU-scoring follow-up

0.2.7's squash (9ea0449) carries the serving half onto main; the GPU scoring kernel is a separate change and is not in it. That kernel is @envy-ai's work, opened as #2 in my fork (xiezhipeng-git#2). I rebased it onto 5e8a4cc as gpu-logprobs-on-main (https://github.com/xiezhipeng-git/Strata/tree/gpu-logprobs-on-main) with the authorship intact: two commits, 9 files, +550/−1, so the diff against 0.2.7 is exactly those nine files and nothing else. It builds on Windows / CUDA 13.3 / sm_89 here, and logprobs_parity passes against the double-precision CPU reference at 3e-5.

Fresh same-day A/B, three runs per cell, medians; top_logprobs: 20, same model and settings:

engine	without logprobs	with logprobs	added cost
released 0.2.7	6.196 ms/token	7.827 ms/token	1.631 ms/token
0.2.7 + GPU scoring	6.365 ms/token	6.870 ms/token	0.505 ms/token

The useful comparison is the added cost within each row, not the difference between the two no-logprobs baselines. The first plain request on each engine is cold (9.6 and 9.4 ms/token against ~6.0 afterwards); dropping it moves the two added-cost figures to 1.75 and 0.57 ms/token, so the direction does not depend on that sample. An earlier run with a different bench harness, on the same card and over the same two variants, is in the #2 thread linked above and gives the same direction, 1.46 → 0.17 ms/token.

Tests

The same discovery command CI uses, on each tree and each platform:

tree	where	tests	result
0.2.7 (5e8a4cca)	Windows, zh-CN	1217	1 error, 26 skipped
0.2.7 + GPU scoring (3a34eb21)	Windows, zh-CN	1218	1 error, 26 skipped
0.2.7 (5e8a4cca)	Linux, T4 ×2 (sm_75)	1217	OK (skipped=14)
0.2.7 + GPU scoring (3a34eb21)	Linux, T4 ×2 (sm_75)	1218	OK (skipped=14)

The one error is already in 0.2.7 and has nothing to do with logprobs: test_winjob.test_children_end_with_a_killed_server. It is upstream's, and it is still unfixed in 0.2.7 — a test-only fix was opened as #3017, was closed with the "0.2.0-beta tracker reset", and was reopened as #3240, which is open today. It reproduces without any Strata code: subprocess.run(["tasklist"], capture_output=True, text=True) on a zh-CN Windows box with PYTHONUTF8=1 returns stdout=None, because tasklist writes the console's OEM code page.

On our side it is visibly not this change: a clean 0.2.7 checkout with no local modifications fails identically, serve/test_winjob.py and serve/winjob.py are byte-identical in 0.2.7 and in both branches here, and with PYTHONUTF8=0 that file is 3/3 green. The extra test in both GPU rows is the one that patch adds.

The two Linux rows are the measurement behind that, not an inference from it: the Windows-only tests skip there and test_winjob errors on neither tree, which is how the release carries it through CI. No job in this repo compiles CUDA or HIP at all, so every CUDA build is hand-run; both Linux rows are sm_75 on CUDA 12.8, each tree built from its own commit, on the 0.2.7 base.

Still on the CPU scoring path: allowed_token_ids (the named path reads the full row per named id instead of a top-K, then renormalises onto that set), vocabulary/K outside the fixed GPU buffers, and the existing SYCL / Metal / gfx906 paths. Wave32 HIP remains unrun.

[leonardomurillo]

Thanks for porting it. I tried 0.2.7 on three Ada cards, an RTX 4090, an RTX 4080 and an RTX 4080 SUPER (Windows, the CUDA 13 engine package 0.1.52, driver 610.88). All three ran Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS with the same engine arguments as our own build and --adapt-swaps 0, one request at a time.

What works, on all three cards:

• logprobs + top_logprobs: 5 with max_tokens: 1: entries come back (B −0.0431, then B, A, … on the fruit prompt from [Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059).
• allowed_token_ids with two ids: the line is renormalised over the named set. For example, −0.0920 and −2.4318 sum to 1.000 in probability.
• 116 requests in a row with allowed_token_ids, then 8 identical requests at 92 and at 775 prompt tokens: no errors, and the 8 repeats are bit-identical within one server session.

One thing to know when comparing numbers: across a server restart, the same request gives a different logprob, e.g. −0.1382, −0.1023 and −0.1294 at 92 tokens in three sessions on the 4080. With STRATA_PREFILL_CPU_SHARE=0.5 both sessions gave the same value to six decimals (−0.095756 twice). That matches what #4059 found on 0.1.43; the details are there.

Speed, compared with our 0.1.43-based build:

• Decode, 4 prompts × 256 tokens: 4090 93.0 vs 95.0 tok/s (−2 %), 4080 63.0 vs 69.3 (−9 %; that card was noisy, with one run of each build off by about 10 %), 4080 SUPER level.
• Prefill: level at about 2.8k tokens, and 0.2.7 is 1–2 % faster at about 11k.

Nothing I can tie to the logprobs code, which isn't used by plain chat requests.

[xiezhipeng-git]

Thanks for porting it. I tried 0.2.7 on three Ada cards, an RTX 4090, an RTX 4080 and an RTX 4080 SUPER (Windows, the CUDA 13 engine package 0.1.52, driver 610.88). All three ran Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS with the same engine arguments as our own build and --adapt-swaps 0, one request at a time.

What works, on all three cards:

• logprobs + top_logprobs: 5 with max_tokens: 1: entries come back (B −0.0431, then B, A, … on the fruit prompt from [Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059).
• allowed_token_ids with two ids: the line is renormalised over the named set. For example, −0.0920 and −2.4318 sum to 1.000 in probability.
• 116 requests in a row with allowed_token_ids, then 8 identical requests at 92 and at 775 prompt tokens: no errors, and the 8 repeats are bit-identical within one server session.

One thing to know when comparing numbers: across a server restart, the same request gives a different logprob, e.g. −0.1382, −0.1023 and −0.1294 at 92 tokens in three sessions on the 4080. With STRATA_PREFILL_CPU_SHARE=0.5 both sessions gave the same value to six decimals (−0.095756 twice). That matches what #4059 found on 0.1.43; the details are there.

Speed, compared with our 0.1.43-based build:

• Decode, 4 prompts × 256 tokens: 4090 93.0 vs 95.0 tok/s (−2 %), 4080 63.0 vs 69.3 (−9 %; that card was noisy, with one run of each build off by about 10 %), 4080 SUPER level.
• Prefill: level at about 2.8k tokens, and 0.2.7 is 1–2 % faster at about 11k.

Nothing I can tie to the logprobs code, which isn't used by plain chat requests.

Thanks — three Ada cards, 116 consecutive allowed_token_ids requests, and bit-identical repeats within one session is exactly the coverage this needed.

The cross-restart logprob differences are consistent with the documented behavior of the automatic CPU prefill share, rather than seed-based randomness or an LP-readout bug. The LP value is computed from the head logits before sampling. With STRATA_PREFILL_CPU_SHARE unset on a qualifying CUDA single-GPU server, auto measures CPU and GPU expert times and adjusts the per-layer split using running means. A new server process can therefore assign a different set of experts to the CPU, and the CPU and GPU arithmetic need not produce identical last bits. Those differences propagate through later layers to the logits.

Your fixed-0.5 repeats support that explanation. They do not isolate every possible contributor, so I would not attribute the remaining delta to --expert-cache auto or to a router flip without another controlled A/B. These results do not show an error in LP line ownership, token alignment, or allowed_token_ids renormalisation.

For strict numerical comparisons, use the same build and hardware and fix the adaptive execution paths. In this recipe that means --adapt-swaps 0 plus STRATA_PREFILL_CPU_SHARE=0 or a fixed value such as 0.5, with the same cache and prefill settings. =0 removes the CPU-share path; a fixed value removes the timing-based share selection but still uses mixed CPU/GPU arithmetic. The startup message's mean KL of about 0.004 is a previously measured distribution-level figure, not a live measurement or a per-token bound, so it cannot be compared directly with one token's roughly 0.04-nat change. The broader reproducibility investigation remains in #4059.

I also agree that the speed comparison should not be attributed to the logprobs field. A plain chat request does not enter the LP-readout path: the server adds logprobs= to the GEN line only when requested, and without it emit_lp_line returns on its initial k < 0 check. The 0.1.43-to-0.2.7 decode difference is therefore a version/build comparison, not a measured cost of this field; the noisy 4080 runs should not be read as a regression on their own.

For reference, the same-card A/B for the field itself on this machine was:

engine	without logprobs	with top_logprobs: 20	added cost
released 0.2.7, CPU scoring	6.196 ms/token	7.827 ms/token	1.631 ms/token
0.2.7 + the separate GPU-scoring follow-up	6.365 ms/token	6.870 ms/token	0.505 ms/token

Those are medians of three runs per cell and measure the cost of requesting logprobs, not the 0.1.43-to-0.2.7 difference. The GPU-scoring follow-up is not in the 0.2.7 release. If the cross-version decode gap is investigated further, the 0.1.43 source build and the released 0.2.7 package should also be treated as a packaging/build-flag comparison, not assumed to differ only by version.

[gdevenyi]

Closing the loop on this one from my side, now that 0.2.7 carries it (thanks @xiezhipeng-git, and @Niko1221 for the release).

Tested on current main (46c152a1, after 0.2.7.1), engine 0.1.53 built from source, Qwen3.8-Flash-Next IQ3_S, on an RTX 5080 (sm_120, CUDA 13.4) and a GTX 1080 Ti (sm_61, CUDA 12.9, -DSTRATA_EXPERIMENTAL_SM60=ON):

• Chat logprobs: streamed and not, with no max_tokens, with max_tokens: 0 and 5000, and with a json_schema: all 200, and the entries spell the reply. This matches 4b7339e1's no-cap rule.
• --parallel 2 (the part you noted was unexercised on my side): a max_tokens: 1 request with logprobs is answered through batch admission, on chat and on /v1/completions (with [Other] serve: logprobs on /v1/completions (stacks #3332) #4056). Two overlapping one-token completions each got their own row (B / A on a fruit / vegetable pair). max_tokens: 2 is refused with the "parallel" message.

Your point about one rule for both routes: #4056 is rebased onto main and takes 0.2.7's checks as they are, shared by chat and /v1/completions through one check_logprobs(), with no cap of its own. Its one deliberate difference is that it refuses logprobs with stream: true. It also reads a batch admission's _strata_lp_lines, as LogprobEntries.take does.

Two follow-ups build on these LP lines:

• [Engine] canonical reads: a typed decision gets the same logprobs for the same prompt (#4059) #4604, canonical reads: a one-token logprobs request gets the same values for the same prompt, whatever the expert cache holds (details in [Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059).
• New: StrataBackend (draft, needs Strata logprobs on /v1/completions) nokia-applied-research/AnyJev#20 / kv: add Q4_0 KV cache mode with FWHT-256 Hadamard rotation #21: a decision backend and gateway reading them.

[xiezhipeng-git]

Closing this as landed elsewhere. 0.2.7 carries the feature, and @gdevenyi's last run (main 46c152a1, engine 0.1.53, sm_120 and sm_61, --parallel 2 included) closes the one gap this branch still had open.

Recording why this branch is not the shape that landed. The engine half here is a dedicated logprobs kernel (src/kernels/cuda/logprobs.cu, include/strata/kernels/logprobs.hpp and the logprobs_parity target), authored by @envy-ai in a2ea8f95 and 7265fc75, and it has no counterpart in main. Main builds the same LP lines through Verifier::window_logprobs (src/core/verify.cpp:3543) and emit_lp_line (src/program/generate.cpp:170), reusing the verify window's head logits. The rest of this PR is in main as it stands: serve/server.py, serve/structured.py's extract_json, docs/DETAILS.md's bullet, serve/test_logprobs.py and serve/test_logprobs_parallel.py. So a rebase would add a second implementation of the same lines rather than finish this one, on a branch now 564 commits behind.

If the kernel path is worth having on its own, it belongs in a fresh PR against current main, with numbers against window_logprobs on the cards it is meant to help.

The validation record continues on #4056 (logprobs on /v1/completions, one shared check_logprobs(), no cap of its own) and #4604 (canonical reads). Thanks @leonardomurillo for the own-lines fix this started from, and @gdevenyi for the hardware runs.