Repository navigation
serve: a prompt's last call runs its last id as the call's last row on the Qwen3.8 seat - #1025
Conversation
…n the Qwen3.8 seat The server fed a prompt less its last id, then stepped that id. `Engine::prefill_next` hands the engine the last call and that step as one call; its default is the two calls as before. The Qwen3.8 seat overrides it where the last id rides the ubatch walk (`GpuModel::closes_by_ubatch`): the body's closing call (`Body38::set_close`) runs the id as the last row of the last ubatch, whose head gives the argmax, and seals the end checkpoint before it, each delta carry given back that row. When that id completes a delta chunk, the walk's last unit keeps the chunk out of the lane (`enqueue_delta_prompt`'s `keep`), so a delta carry now holds up to a whole chunk, 16 rows (`DELTA_CARRY_ROWS`), and the next step folds it. A 4,096-id prompt now folds too. The carry is 33,152 B more a GDN layer, 1,193,472 B over Qwen3.8's 36, which moves the placement pins and fits 28 checkpoints in the host budget where 29 fit. A call of passes, a last ubatch that would hold the id alone, and the MTP draft on more than one slot keep the call less the id and the id's step: a slot change clears the draft's waiting row in the ubatch arena. Under steps the closing call now feeds its last id after the end mark. gate_linear's clause 16 holds a kept call and the calls after it to the one call bit for bit, and a step on the kept carry to the one-lane rule; the checkpoint folder gate's (cl) clause holds the closing call to the one call's first token, the cut call to the whole call and the step from the end mark to the former cut.
github-actions
Bot
added
gpu
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: CLEAN: No validated must-fix items remain.
Non-blocking
⚠ Complete the performance evidence: qwen38.rs:3324. The PR supplies gate output, a hardware table and timing summaries, but omits the nvidia-smi line and exact timed A/B commands with output required by CONTRIBUTING.md. Attach these for both measured commits to support the changed prompt dispatch's speed claim.
Checked
Duplicates: dryer reports zero qualifying pairs in its searched crates; GPU crates were excluded from that search.
Complexity: diagnostics flag new gate helpers and closing-checkpoint branches. Complexity alone does not establish a defect; the kernel's added keep condition does not warrant a split.
Silent failure: closing-call and checkpoint-seal errors propagate; the prompt launcher checks buffer sizes.
Tests, static inspection: keep_clauses checks kept output, state and carry; close_case checks rollback equivalence; the new serve test asserts that prefill_next receives the whole prompt.
Gates: changed carry and placement pins include dated derivations.
Escape catalog: C1–C4 do not match the cited Rust dispatch path.
Notes
Static validation only; builds and hardware gates were not rerun. gpu.json records evidence: false for the incomplete evidence.
Card check: needs-gpu. This pull request changes crates/gpu-gates/src/bin/gate_linear.rs, crates/gpu-gates/src/bin/gate_qwen38_serve.rs, crates/gpu-gates/src/bin/gate_qwen4exp_e2e.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.
🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)
|
The review is done and the contribution check passed, so I'm merging this. |
Qwen3.8의 프롬프트 마지막 호출이 마지막 id를 마지막 행으로 함께 처리해 prefill과 TTFT가 약 2% 빨라집니다.
What and why
The server fed a prompt less its last id, then ran that id as a separate step.
Engine::prefill_nextnow hands the engine the prompt's last call and that step as one call; its default is the same two calls as before. The Qwen3.8 seat overrides it: the last id runs as the last row of the call's last batch, whose head gives the first token, and the end checkpoint is sealed before it.When that id completes a delta chunk, the call keeps the chunk out of the lane, so a delta carry now holds up to 16 rows (
DELTA_CARRY_ROWS) and the next step folds it. A 4,096-id prompt takes this path too. The carry is 33,152 B larger per GDN layer, 1,193,472 B over Qwen3.8's 36 layers, which moves the placement pins and fits 28 checkpoints in the host budget where 29 fit.A multi-pass call, a last batch that would hold the id alone, and the MTP draft on more than one slot keep the old two calls.
Release notes: Changed: Qwen3.8 prefill and time to first token are about 2% faster on a 4,096-token prompt.
Code shape
seal_end_mark_with(crates/gpu/src/arch/qwen3moe/body38.rs:3320) with the end mark's.Enginetrait gained one method whose default is the old two calls.fold_closesand its carry-header read are removed.PIN(2026-10-30):and its derivation; no band or tolerance moved.cargo fmtandcargo clippyare clean on what I touched; the lint count did not go up.Duplication found or removed (
path:line): none left; the copy of the end mark's seal was merged intoseal_end_mark_with.What I ran
Speed,
bloomery-serveon Qwen3.8-Flash-Next-FP4-Local: a 4,096-id prompt, 128 generated tokens, 8 windows, three rounds after one warm-up, draft off and MTP on. Base5c9f5d78fagainst headbab953d55, both built from source.Prefill and TTFT gain in every arm with the ranges apart. Decode shows no resolved change. The server log shows 1,193,472 B more resident memory on head, the larger carry. In the MTP arm on code-fn the greedy ids first differ at token index 12 of request 1 (6563 on base, 1715 on head), since the first token now comes from the last row of the batch instead of the separate step.
Kernel:
gate_linearclause 16 holds a kept last chunk, the calls after it and a step on the 16-row carry to the one-call result.Model: the closing call against the one call, on Qwen3.8-Flash-Next-FP4-Local. Before this change the 32-id case ended with
the last ubatch 1 rows (folds false); after it:Placement pins and the end-to-end gate:
The carry-size derivation in
gate_qwen4exp_e2ealso matches the store on the real tier:structure layers=48 ... store bytes 237766352 (want 237766352, derived) PASS.Hardware (if the change is for a machine)
The speed A/B ran on the GB10; the end-to-end gate ran on the RTX 5090.
The kernel and model gates ran on DGX Spark machines of the same kind as the A/B.
AI help
AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.