Repository navigation
Conversation
This was referenced Oct 25, 2026
|
Tested on hardware: merged with #3200 onto
#4056 adds logprobs (#3200) on this route, for decision clients that read vLLM's legacy |
Niko1221
pushed a commit
that referenced
this pull request
Oct 27, 2026
…etions (#3200) /v1/chat/completions on the CUDA and HIP engine accepted logprobs and returned an empty choices[0].logprobs.content. Now "logprobs": true with "top_logprobs" 0 to 20 (and the legacy "logprobs": N) returns one entry per token with its alternatives, streamed or not, and "allowed_token_ids" returns a distribution renormalised over exactly the ids the caller named. Engine (src/program/generate.cpp): the GEN line takes logprobs=K and allowed=a,b,c. After each "T id" the engine writes one "LP <logprob> [id:logprob ...]" line from the verify window's head logits before sampling, penalties and temperature. The SYCL engine already wrote LP lines for logprobs=K; it now reads allowed= as well, with the same line format. A request without logprobs reaches none of this: both call sites are behind "req_logprobs >= 0", a local int of the request, so the default path prints what it printed before and does no extra work per token. Changes from the PR: the call sites are guarded instead of calling the function for every token, the function is static, the token index is checked against the row, and the Chinese comment is in English. Server (serve/server.py): LP lines are read before the token is yielded and belong to the request that asked (tagged with its cancel Event), the entries spell the reply, json_schema replies return no logprobs rather than a mismatching set, an allowed id past the vocabulary is a 400, and logprobs on a --parallel server is answered for max_tokens 1 only (a slot's BT lines carry no LP line). serve/structured.py: _extract_json is now extract_json (the logprobs alignment uses it); the two uses added by the #2893 retry follow the rename. Left out: the PR's bench and result files (none in the diff). #3332 (/v1/completions) and #4056 (logprobs on it) build on this change; this one needs nothing from them. Tests: serve.test_logprobs and serve.test_logprobs_parallel (30 tests with the PR's), serve.test_structured, serve.test_responses: 94 tests OK. Not covered: no model ran here. The PR author reports 28 of 28 on an RTX 5080 with MTP and on a GTX 1080 Ti; the layer-split pipelined decode path (the first call site) was not run by anyone, and the LP values were not compared with a CPU softmax of dumped logits. Co-authored-by: Leonardo Murillo <4869242+leonardomurillo@users.noreply.github.com>
|
Update to my test above: this PR still merges cleanly onto current #4056 now stacks on this PR alone, since #3200 shipped in 0.2.7. It adds logprobs, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result
Clients that build their own prompts need
/v1/completions, but Strata has no legacy completions endpoint. Sending those prompts through the chat API would apply the chat template again.This adds
/v1/completionsfor string prompts, with streaming and non-streaming responses in the OpenAItext_completionformat, including token usage.Related discussion
#2277, #2278.
This replaces #2278 after the tracker reset. The
/reopenformat check passed twice, but GitHub returned "Validation Failed" both times. The bot suggested opening a new PR and linking the old one. The earlier review and fixes are in #2278.Approach
The handler reuses the existing generation path without rendering a chat template.
max_tokens: 0or a negative value uses the remaining context.If the caller explicitly disables thinking, the handler appends the template's no-thinking tail. Otherwise it leaves the prompt alone.
Streamed text is sent once. With
echo: true, the prompt goes out once at the start, and the finish chunk has empty text. The tests concatenate the chunks and compare them with the non-streaming reply, with echo on and off.The structured-output error handler stays in the chat endpoint. A failed streamed response sends an SSE error followed by
[DONE], rather than a second HTTP response inside the stream. Engine errors after streaming starts are also returned as SSE errors.Validation
Tested
7e08755f, rebased onto maine88ee39d(0.2.0-beta).Passed:
On Windows with Python 3.13.2 and jsonschema installed:
368 tests, OK, 3 skipped.
On Linux in a Python 3.11 Docker container, using a dedicated venv with only the pinned
requirements.txtpackages:The smoke test reported
"passed": true.git diff --check upstream/mainpassed too.Failed:
In the same Linux venv:
python -m unittest discover -s serve -p 'test_*.py'888 tests, 2 failures, 18 skipped. The failures were:
test_lifeline.Lifeline.test_the_portable_watcher_ends_the_child_when_the_server_is_killed: the test still sees the child after killing the server.test_server.VisionTempFiles.test_work_dir_has_no_space_when_temp_has_one: the temporary path contains a space.I reran both tests on clean main
e88ee39dwith the same container and dependencies. Both failed with the same assertions there. This PR doesn't change either path.Not run:
Limits
The endpoint accepts one string prompt and supports only
n=1andbest_of=1. Suffix and logprobs are rejected.stream_optionsaccepts onlyinclude_usage, with streaming enabled. Usage is still on the finish chunk rather than a separatechoices: []chunk, so the OpenAI streaming-usage format isn't fully implemented.