[xiezhipeng-git]

Problem and result. /v1/chat/completions on the CUDA/HIP engine accepted logprobs / top_logprobs and returned HTTP 200, but choices[0].logprobs.content was always []. A client that ranks candidate tokens from the model's own distribution — constrained decoding, candidate scoring, tool-choice re-ranking — silently got nothing: the request succeeds, the content is correct, only the field is missing, so the failure is invisible unless the caller asserts on the field. After this change, logprobs: true with top_logprobs 0..20 (the legacy completions form logprobs: N is also accepted) returns one entry per token with its top alternatives, and allowed_token_ids returns a distribution renormalised over exactly the ids the caller named.

Related discussion. #1916 proposed the same feature for CUDA/HIP and is closed. This implementation follows the protocol the SYCL port already speaks (835d30b) and the Metal port (31f2668), so one server-side reader serves all three backends.

Approach.

• Engine half, src/program/generate.cpp: logprobs=K and allowed=a,b,c are two extra keys on the --serve GEN line (an engine that predates them skips them as unknown), and one LP <logprob> [id:logprob ...] line is written after each T <id> in the same flush. The value comes from the verify window's head logits before sampling, penalties and temperature, so it is the model's distribution rather than the sampler's. Off by default: a request without logprobs produces the previous output byte for byte, and the per-token cost is one integer compare.
• Server half, serve/server.py: sampling_keys appends the two keys only when asked; generate reads each token's LP line before yielding the token (a held line is consumed before the next read, so an engine without LP lines loses no lines, and a DONE read while looking for one still ends the request); LogprobEntries decodes token bytes incrementally so the entries joined spell the reply; allowed_token_ids keeps the named ids and renormalises over them; structured_chunks re-aligns the entries when response_format: json_schema rewrites the content and returns no logprobs rather than a set that does not match.
• Compatibility: additive and opt-in. A request that does not ask for logprobs gets the same bytes as before, and no default changes.

Validation.

Rebased onto upstream/main e88ee39d (which contains v0.2.0-beta b73dbce3); head 104becfd. The diff is 3 files, +570/−8, unchanged by the rebase.

Passed:

• Full CUDA build and link, revision 104becfd: cmake --build build-lp --target strata — Release, CMAKE_CUDA_ARCHITECTURES=89, CUDA 13.3 (V13.3.73), MSVC 14.41, Windows 11, RTX 4090. [154/154] Linking CXX executable strata.exe, exit 0.
• python -m unittest serve.test_logprobs — 17 tests, OK, on 104becfd.
• Hardware run, same revision and engine (strata.exe built from it, model Swift-1.5 IQ3_XXS, --spec 4 --spec-min-p 0.5 --kv int8 --kv-resident 32768), max_tokens: 1, temperature: 0, chat_template_kwargs.enable_thinking: false:
  • non-streamed, top_logprobs: 20 → logprobs.content length 1, top_logprobs width 20; entry token A at -0.1028 with B -3.121, C -3.201, D -5.1109.
  • the same request with tools and tool_choice: "auto" → entries present, width 20, labels A/D/C/B.
  • stream: true → the delta carries logprobs.content (1 entry, token A, -0.0801, top width 20).
• An external client (a separate project) that scores 5 candidate labels from this endpoint: 7 scoring requests in a 30k-token session, 0 errors, 0 disabled calls. Run against the same commit before the rebase (80494788); the endpoint behaviour it depends on is the same code path exercised above.

Failed: none observed.

Not run:

• HIP backend build. src/program/generate.cpp is shared by the CUDA and HIP builds; only the CUDA configuration was built here (no AMD GPU or toolchain on this machine).
• SYCL backend build. It carries its own implementation (install_logprobs in sycl/serve/server_intel.py) and this change does not touch it; no Intel GPU or toolchain here.
• --batch mode: see Limits.

Limits.

• --batch does not return logprobs. generate_batched never reads LP lines; only the solo/serial path (generate) does, so a server started with --batch answers logprobs.content: []. Giving the batched path the same read-before-yield step is the remaining work if that combination is wanted.
• The reported logprob describes the model's distribution before penalties and temperature, not the post-sampling one.
• With response_format: json_schema the returned content may not be spelled by the tokens; in that case no logprobs are returned, by design.
• CUDA/HIP only; the SYCL and Metal implementations are untouched.

[qiuzan2001]

Some scope notes from the diff: (1) README.md gains a banner "这是 xiezhipeng-git/Strata —— 一个加了 OpenAI API logprobs 的 fork" at the top of upstream's README, and .gitignore is rewritten from LF to CRLF (all 71 lines carry \r, hence +71/-63) with fork-local entries; README.local.md, START-CN.bat, strata_125b.bat and build-win-env.bat are launchers and notes for one install. (2) The engine change is not in the diff as C++: openai_logprobs/build_engine.py patches a copy of src/program/generate.cpp by text anchors at build time, so the C++ cannot be reviewed or built as part of the normal tree, and it covers CUDA only. (3) #1916 (open) adds OpenAI logprobs to the CUDA/HIP engine directly, and sycl/serve/server_intel.py already has install_logprobs, so this overlaps both. python openai_logprobs/selftest.py here ends with "[SHORT] 4 AnyJev 后端 只跑了 0 项（至少该有 3）".

[xiezhipeng-git]

Some scope notes from the diff: (1) gains a banner "这是 —— 一个加了 OpenAI API logprobs 的 fork" at the top of upstream's README, and is rewritten from LF to CRLF (all 71 lines carry , hence +71/-63) with fork-local entries; , , and are launchers and notes for one install. (2) The engine change is not in the diff as C++: patches a copy of by text anchors at build time, so the C++ cannot be reviewed or built as part of the normal tree, and it covers CUDA only. (3) #1916 (open) adds OpenAI logprobs to the CUDA/HIP engine directly, and already has , so this overlaps both. here ends with "[SHORT] 4 AnyJev 后端 只跑了 0 项（至少该有 3）".README.md``xiezhipeng-git/Strata``.gitignore``\r``README.local.md``START-CN.bat``strata_125b.bat``build-win-env.bat``openai_logprobs/build_engine.py``src/program/generate.cpp``sycl/serve/server_intel.py``install_logprobs``python openai_logprobs/selftest.py

Thanks for the review. All three points were correct, and the branch is rebuilt around them — it is now
3 files (src/program/generate.cpp, serve/server.py, serve/test_logprobs.py), rebased onto current
main (296b59d2), force-pushed. Point by point:

1. The README banner, the LF→CRLF rewrite, and the launcher files

The README.md rewrite you saw was an artefact of a stale base, not a change: feat/openai-logprobs was
based on a0116b27 while main had moved 282 commits forward, so the page showed "old base vs branch" —
370 files, and a README that reads as rewritten (17/31) but is one added block against current upstream.
Against 296b59d2 it is 15/0.

.gitignore really was rewritten to CRLF (71/71), burying the eight lines actually added — that one was
mine, and it is fixed: the file is now byte-for-byte upstream, so it no longer appears in the diff at all.

The launcher files (README.local.md, START-CN.bat, strata_125b.bat, build-win-env.bat) are gone
from this branch. They were a bilingual double-click installer for my own use and have nothing to do with
logprobs; they will be their own submission if they are wanted at all.

2. The C++ could not be reviewed or built as part of the normal tree — fixed

That was true of the old version, and it was the most important thing wrong with it: the engine source was
patched into a generated copy at build time and built as a separate EXCLUDE_FROM_ALL target, so it was not
in the tree, not in setup.py's --target strata build, and therefore not in any binary a user installs.

It is fixed in the way your point implies: the C++ now lives in src/program/generate.cpp itself —
logprobs=K and allowed=a,b,c as two request keys on the --serve line, and emit_lp_line(), which
prints LP <logprob> [id:logprob ...] after each T <id> in the same flush. +60 lines, reviewed as normal
tree, built by the normal target, shipped by the normal release. The pipeline path asks V1(A) (the
verifier of the window whose tokens that loop prints) and the serial path ver, matching each token's own
window. Off by default: a request without logprobs gets today's output, byte for byte.

serve/server.py (+248/−8) does the OpenAI side: top_logprobs / allowed_ids translate the request,
generate reads each token's LP line before yielding the token, LogprobEntries decodes token bytes
incrementally so the entries, joined, spell the reply, and structured_chunks re-aligns the entries when
response_format: json_schema rewrites the content — aligned or dropped, never mismatched.

allowed_token_ids is vLLM's semantics — only the named ids, renormalised over them. I would argue it
belongs with the feature rather than after it: without it a client still filters and renormalises by hand,
which is where the answer drifts. It is inert unless the request names ids, so it costs the default path
nothing.

3. install_logprobs is not from #1916 — but #1916 is the real overlap

install_logprobs is upstream 835d30b1 (sycl: OpenAI logprobs on /v1/chat/completions, maxfridbe,
already in main), covering SYCL; this branch reuses its protocol — the LP line format is the same, so
one server-side reader serves both engines. Metal has it too (31f26683). CUDA/HIP was the gap.

#1916 is the real overlap: it does the same engine and server halves, and it is not merged yet. Both are now
complete submissions, so the honest question is which one to stop, and that is yours. Before you weigh them,
the delta — what a client gets from this one that #1916 does not offer:

• allowed_token_ids — vLLM's semantics: the request names token ids, and the reply carries only those,
  renormalised over exactly them. This is the part that makes the distribution controllable: a caller doing
  distribution-based stopping or scoring asks about its own candidate set and gets a distribution over that
  set, instead of a top-K list it has to hope contains the candidates and then filter and renormalise by hand —
  which is where answers drift. Neither OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine #1916 nor the upstream SYCL port has it. It is inert unless the request
  names ids, so it costs the default path nothing.
• json_schema alignment — with response_format: json_schema the content is rewritten by the validator,
  and tokens that spelled the model's answer no longer spell what is returned. Here the entries are re-aligned
  to the validated content when a token run spells it, and dropped otherwise — a client that joins the tokens
  never silently gets a mismatch. OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine #1916 does not handle this case.

And the weigh-up as I see it:

• Keep this one — everything OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine #1916 has (core logprobs on CUDA/HIP, the same LP protocol, streamed and
  non-streamed, thinking tokens included) plus the two above, plus 17 tests in serve/test_logprobs.py
  (python -m unittest serve.test_logprobs, no GPU).
• Keep OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine #1916 and take the rest as a patch on top — then I fold allowed_token_ids and the
  json_schema alignment into a small patch against it, so there is one implementation instead of two. If the
  core lands without allowed_token_ids, a client that needs a distribution over specific candidates still
  cannot get one from a stock install — that is most of what the feature is for, so I would argue against
  stopping here.

4. [SHORT] 4 AnyJev 后端 只跑了 0 项（至少该有 3）

Correct, and worse than a flaky check: that test section imported AnyJev's own
anyjev/backends/strata.py from a sibling checkout, so a clone without AnyJev beside Strata went red for a
reason unrelated to this repo. My first fix (report it as a skip) only hid that; the section is gone, and the
launcher it belonged to is out of this PR entirely.

Same pass, a second bug: the scripts printed ✓/✗ while the Windows console defaults to GBK, so they died on
their own output with UnicodeEncodeError — the generator crashed on the success line printed after it
had already written the files.

Verification

• python -m unittest serve.test_logprobs — 17 tests, OK, no GPU: the request keys, the LP read
  (including the post-STOP drain), the reply shape byte for byte, streaming, thinking's two parts each
  spelled by its own tokens, allowed_token_ids renormalised to 1, the schema cases.
• Upstream suites unchanged: serve.test_server 314 OK, test_structured 10, test_responses 40,
  test_logit_bias 5, test_security 23 — 392 tests.
• End to end through the stock server: no logprobs requested → unchanged output; logprobs on → entries
  that spell the reply; streamed; allowed_token_ids → only the named ids, summing to 1; json_schema →
  54 entries whose tokens spell the validated content; an engine without LP lines → no logprobs, reply
  unchanged.
• Performance: the two submissions should be identical on the engine side (same max, logsumexp,
  partial_sort, one printf per token); the server side differs by ~0.8 µs/token in where the LP line is
  read, under 0.01% of wall time at the 33–102 tok/s in docs/DETAILS.md. I would rather you measured it
  than took my word — same machine, same model, logprobs actually on; tools/strata_bench.py does not
  send logprobs, so the request needs extending.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[xiezhipeng-git]

/reopen

[github-actions]

The check passed, but GitHub would not reopen it (Validation Failed: ). If the branch was deleted, restore it or open a new pull request and link this one.

[xiezhipeng-git]

Rebased onto main (which contains v0.2.0-beta), retitled with the [Engine] prefix, and the template filled in. GitHub would not reopen this one because the branch was rebased (state cannot be changed. The feat/openai-logprobs branch was force-pushed or recreated.), so the same single commit continues in #3200.