[gdevenyi]

Problem and result

Decision clients such as AnyJev render their own chat prompt and read the label tokens' log-probabilities from vllm serve's legacy /v1/completions. Strata 0.2.7 has logprobs on /v1/chat/completions (#3200, landed as 9ea04496), and #3332 adds /v1/completions but refuses logprobs. So a client can have its own prompt or the logprobs, but not both.

This PR puts the 0.2.7 logprobs on #3332's route. /v1/completions takes logprobs: N (or true with top_logprobs), allowed_token_ids and return_tokens_as_token_ids, and the prompt can be a list of token ids. It returns vLLM's legacy logprobs object.

 POST /v1/completions                         (#3332)
   prompt: str
+  prompt: [token ids]                         # vLLM form; one prompt per request
-  logprobs → 400 "logprobs are not supported"
+  logprobs: N | true + top_logprobs           # GEN … logprobs=K, as chat
+  allowed_token_ids: [ids]                    # full-vocabulary values, not renormalised
+  return_tokens_as_token_ids: true            # keys "token_id:<id>"
 → choices[0].logprobs
+    {tokens, token_logprobs, top_logprobs: [{tok: lp}], text_offset}

Related discussion

• Clients: New: StrataBackend (draft, needs Strata logprobs on /v1/completions) nokia-applied-research/AnyJev#20 (StrataBackend, which reads this route with allowed_token_ids and return_tokens_as_token_ids) and New: anyjev.serve --backend strata, the gateway's Decider mode (stacks on #20) nokia-applied-research/AnyJev#21 (its HTTP decision gateway, anyjev.serve --backend strata, one gateway per Strata server). The scope discussion on the AnyJev side is Strata backend (anyjev/backends/strata.py) and a Decider mode for the gateway: scope check (#20, #21) nokia-applied-research/AnyJev#22.
• Depends on [Other] add legacy /v1/completions for raw-prompt clients (#2277) #3332 only. This branch is main (46c152a1, after 0.2.7.1) + [Other] add legacy /v1/completions for raw-prompt clients (#2277) #3332 merged unchanged (clean merge) + three commits, b6132c9a..d032be99 (+216/−20 in serve/server.py, serve/test_logprobs.py, docs/DETAILS.md). When [Other] add legacy /v1/completions for raw-prompt clients (#2277) #3332 lands, it is just these commits.
• [Engine] OpenAI logprobs / top_logprobs / allowed_token_ids for the CUDA/HIP engine #3200 shipped in 0.2.7, so the stacking on it is gone. This branch was rebased onto main instead of re-merging [Engine] OpenAI logprobs / top_logprobs / allowed_token_ids for the CUDA/HIP engine #3200's branch.
• Originating request: Feature request: logprobs / top_logprobs on /v1/chat/completions (typed decisions from one forward pass) #675. Related: [Engine] typed decisions: score a schema's token paths instead of generating them #3743 (a scored typed-decision endpoint; this PR gives existing vLLM-style decision clients the field they already read). Repeatability: [Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059.

Approach

• The same rules as chat, in one place. check_logprobs(req, svc, max_req) holds 0.2.7's two checks, now shared by both routes:
  • --parallel refuses logprobs unless max_tokens is 1.
  • A named id outside the vocabulary is a 400.
  • Like 0.2.7, there is no length cap of its own (@xiezhipeng-git's point on [Engine] OpenAI logprobs / top_logprobs / allowed_token_ids for the CUDA/HIP engine #3200: one way, said in the text). The earlier 1..4096 bound is gone from this branch.
• CompletionLogprobs. It reads only the LP lines tagged with this request's cancel Event, the ownership rule 0.2.7's LogprobEntries uses. That includes the lines a batch admission keeps on the Event (_strata_lp_lines). It lists every generated token, the stop token too, so a one-token read whose top token is <|im_end|> still has its row.
• No renormalising. allowed_token_ids reports the named ids' full-vocabulary values. For those ids these are the numbers vLLM's default raw_logprobs gives. A decision client checks how much of the mass its labels hold, and renormalised values always sum to 1. Chat keeps 0.2.7's renormalised shape. Both are documented in docs/DETAILS.md.
• One limit of this route: logprobs with stream: true are refused (400) to keep the change small.
• Compatibility. No engine change. Requests that don't ask for logprobs are unchanged.

Validation

d032be99, engine 0.1.53 built from this branch, with Qwen3.8-Flash-Next IQ3_S and the args strata launch writes (--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp … --kv int8 --ple-io ram --kv-resident 32768, no --vision). Two machines:

	epic	monster
GPU	RTX 5080 16 GB, sm_120	GTX 1080 Ti 11 GB, sm_61
Build	CUDA 13.4	CUDA 12.9, -DSTRATA_EXPERIMENTAL_SM60=ON, g++-14
OS	Linux	Linux (CachyOS)

Passed:

• python -m unittest discover -s serve -p 'test_*.py' (as CI runs it): 1,234 tests, 2 failures.
  • GpuChoice.test_hip_ordinal fails on main here too.
  • LiveRate.test_the_live_number_is_a_rate failed once while the machine was under memory pressure; it passes 3/3 on this branch and on main when re-run.
• serve.test_logprobs: 41 OK. That is main's tests plus 11 for completions:
  • the legacy shape, the chat form of the field, echo offsets;
  • un-renormalised allowed_token_ids, token-id keys, token-id prompts;
  • logprobs: false, no length cap, the refusals;
  • two overlapping requests that each read only their own LP lines.
• Hardware, 12 checks on both machines:
  • chat logprobs, streamed and not, with no max_tokens and with a json_schema;
  • completions shape, allowed_token_ids raw values equal to the top-K read, token-id prompt, max_tokens: 0, logprobs: false;
  • the stream refusal.
  • With --adapt-swaps 0 and STRATA_PREFILL_CPU_SHARE=0.5: 12/12 on both. Chat, text-prompt completions and token-id completions give the same values to 6 decimals (epic B −0.027356, monster −0.035613).
  • With the launch defaults: 11/12 on both. The one miss is that chat-vs-completions value comparison, because reads drift between requests there ([Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059), not because the routes differ.
• --parallel 2 on epic: completions max_tokens: 1 with allowed_token_ids → 200 with both ids. max_tokens: 2 → 400 naming --parallel. Two overlapping one-token reads (fruit/vegetable) each returned their own answer (B, A) through batch admission.
• AnyJev's StrataBackend (New: StrataBackend (draft, needs Strata logprobs on /v1/completions) nokia-applied-research/AnyJev#20), on epic with --adapt-swaps 0 and STRATA_PREFILL_CPU_SHARE=0.5: engine smoke test 7/7. Its parity check (named-id read vs the server's own top-20) gives max_abs_diff 0.000000 in two runs, and the Decider at L0 decides 5 states × 3 questions in 14 s.

Failed:

• None beyond the pre-existing test_hip_ordinal.

Not run:

• HIP, SYCL and Metal builds. No C++ changes here, but this route ran on CUDA only.
• --parallel on monster (11 GB).
• The full suite on Windows.

Limits

• Repeatability is the engine's ([Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059). With the launch defaults, identical requests drift by a few hundredths of a nat. --adapt-swaps 0 --pcie-frac 0 with STRATA_PREFILL_CPU_SHARE=0.5 makes them identical, the first run after a start included. Without --pcie-frac 0 that first run still differs. The recipe costs about 38% of decode speed on an RTX 5080 (none for one-token reads); the measurements are on [Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059.
• One prompt per request (no batched prompt: [[…], […]]). AnyJev's Tacit(engine="server") client batches, so it still needs its own path; its Decider backend (New: StrataBackend (draft, needs Strata logprobs on /v1/completions) nokia-applied-research/AnyJev#20) sends one prompt per request.
• Logprobs are not streamed on this route.

Merge danger

Door: two-way. It is server-only Python behind an opt-in request field.

Blast radius: small. Callers of /v1/completions that send logprobs used to get a 400 and now get logprobs. Chat logprobs go through the moved, otherwise unchanged checks.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Cix1XJhNhNNuKXi26KvXnU

[gdevenyi]

Hardware rerun at e0eb04e6 (with #3200's eca6513) on two machines:

• epic: RTX 5080, sm_120, CUDA 13.4.
• monster: GTX 1080 Ti, sm_61, CUDA 12.9 with -DSTRATA_EXPERIMENTAL_SM60=ON.

Both ran Qwen3.8-Flash-Next IQ3_S with --spec 4 --mtp … --kv int8, --parallel 1. The checks gave the same results on both machines:

• /v1/completions with logprobs: 5 returns tokens / token_logprobs / top_logprobs / text_offset.
• allowed_token_ids returns the same raw values as the top-K read.
• A token-id prompt (return_tokens_as_token_ids) gives the same 28 tokens as the text and chat prompts.
• Refused with a 400: stream + logprobs, max_tokens: 0 + logprobs, and streamed logprobs without max_tokens. This route keeps the bound, unlike chat after eca6513.
• logprobs: false is answered with logprobs: null.
• Chat, text-prompt completions and token-id completions agree to 6 decimals when read back to back. Because of [Engine] identical temperature-0 requests give different logprobs on ~100-token prompts, even with the #410 switches #4059 / Greedy output varies between server requests, but not between fresh engine runs (STRATA_IQ_MT_MIN=1, 0.1.32) #410, reads further apart do not always agree: on monster even 8 identical requests drift every few reads.
• serve.test_logprobs: 39/39. Full serve/ suite: 1,080 tests, with the 1 pre-existing test_hip_ordinal failure that main also has.
• AnyJev's StrataBackend (New: StrataBackend (draft, needs Strata logprobs on /v1/completions) nokia-applied-research/AnyJev#20): engine smoke test passes on both. Decider at L0, 5 states × 3 questions, gave the same decisions on both machines: 15.0 s on epic, 38.6 s on monster.

[ibbobud]

@gdevenyi Really interested in this PR and would love to see this functionality make it upstream!

I'm running Strata with Qwen3.8-Flash-Next on dual Titan Xp GPUs (Pascal, sm_61), and I'm happy to offer some time for testing, benchmarking, or troubleshooting on older NVIDIA hardware.

I also noticed #4059 and would be happy to help investigate whether the logprob consistency issue appears on Pascal.

If there are any tests or validation that would be useful, I'm glad to help however I can. Appreciate the work going into this!

[gdevenyi]

Make sure to express your interest in the interrelated PRs and issues. So far the real blocker is reproducibility.

Ideally I'd like dynamic response to respond properly to certain calls rather than disabling some performance caching features as suggested in the issue.

[gdevenyi]

Rebased onto main after 0.2.7 (#3200 landed as 9ea04496), now d032be99. It stacks on #3332 only. The description is updated.

@xiezhipeng-git, following your point: the length bound is gone. /v1/completions now goes through the same check_logprobs() as chat, with 0.2.7's rules:

• no cap of its own;
• under --parallel, logprobs only for max_tokens: 1;
• a named id outside the vocabulary is a 400.

The one deliberate difference is that this route refuses logprobs with stream: true. It also reads a batch admission's _strata_lp_lines, as LogprobEntries.take does.

Tested on epic (RTX 5080) and monster (GTX 1080 Ti, CUDA 12.9 SM60), engine 0.1.53 built from this branch. Under --parallel 2, two overlapping one-token completions each got their own row. The #4059 matrix on both cards is posted there. With --adapt-swaps 0 and STRATA_PREFILL_CPU_SHARE=0.5, chat and both completions prompt forms give identical values, and AnyJev's parity check is exact.