[0z5a]

Summary

Four fixes to prefill–decode disaggregation, measured on one GH200 node with the prefill engine on GPU 0 and the decode engine on GPU 1:

1. Huginn coordinator config: superseded by [Bugfix][M9] Restore model-neutral PD coordinator metadata #140. This branch used to rebuild the config class from the model_type the worker reports, because the coordinator built every model with OuroConfig. main now resolves the model configuration at the entry point, passes it to the workers and rejects Huginn in PD before spawning them; the merge of main keeps that and drops this change.
2. Coalesced transfers. KV segments contiguous on both sides become one descriptor, and each progress pass fills the in-flight budget instead of sending one chunk per request. The overlap counter now also credits transfers issued during a prefill computation that drained before the next chunk was computed; the wavefront overlap GPU test counted 0 before.
3. VLLM_RLT_NIXL_UCX_ERROR_HANDLING=none|peer creates the UCX backend explicitly with ucx_error_handling_mode (nixl_agent_config cannot pass backend init parameters). With NIXL's default (peer) UCX skips shared memory and cuda_ipc, which have no peer failure handler, and emulates intra-node RMA with active messages over TCP; removing TCP from UCX_TLS then fails at startup. With none, UCX's log shows cuda_ipc and shared memory and no TCP lane. Peer failures are then reported only through the control channel, so the default is unchanged.
4. Default transfer budget: 256 MiB chunks, 2 GiB in flight (was 64 MiB / 256 MiB). Chunks are submitted, and their completion observed, only between engine steps, so a request whose KV exceeds the in-flight budget waits extra steps after its prefill (the commit tail). 2 GiB holds a 2048-token Ouro-1.4B prompt (1.6 GB of KV). In-flight bytes are not a buffer: NIXL writes straight from the registered prefill pool into the decode pool. pd_serve's CLI defaults now read the PDConfig defaults.

stats() also reports prefix-cache hits and queries.

Measurements

Ouro-1.4B 1P1D, 2048-token prompts, 64 output tokens, 32 requests, packed prefill, error handling none, two runs per arm in alternating order:

chunk / in flight	tok/s	TTFT p50	commit tail p50
64 MiB / 256 MiB (old default)	170–279	3.4–3.9 s	0.96–1.05 s
256 MiB / 1 GiB	254–290	3.2–3.5 s	0.68–0.90 s
256 MiB / 2 GiB (new default)	463–466	2.0 s	0.017 s
peer (TCP), 256 MiB / 1 GiB	185–334	3.0–3.4 s	0.74–0.81 s

Switching the link alone (peer → none, old budget, one run each) changed a 64 MiB chunk from 20.0 to 16.9 ms and left the commit tail near 1 s: the budget rounds, not TCP, set the tail. Earlier, behind per-row prefill, chunks four times larger took four times longer under TCP. Only 2048-token prompts were measured; much longer prompts still need several rounds with the 2 GiB default.

Platforms. GH200: torch 2.14.0+cu130, Triton 3.8.0, NIXL 1.4.1. RTX 5090: torch 2.13.0+cu130, Triton 3.7.1. Models: Ouro-1.4B (24 layers, 4 loops, fixed depth) and Huginn-0125 (2 prelude + 4 recurrent + 2 coda layers, 32 loops). Burst workloads issue C requests at once with random-token prompts; session workloads replay multi-turn agent traces with the prefix cache on.

Tests

• CPU suite after merging main: 838 passed; ruff passes. New tests: coalescing preserves every byte and both limits, error-handling parameters, the default budget holds a 2048-token Ouro prompt.
• GPU PD tests on this branch (GH200, head d8fece2, one prefill and one decode GPU, pytest --run-gpu tests/test_pd.py tests/test_transfer_connector.py -k "not flash"): 56 passed; main (66920da) with the same command: 52 passed. Earlier, 41 passed on the combined branch with changes 1–2.

Notes

The GPU numbers come from a development branch that contains all changes of this series; most cells are single runs, and on the shared GH200 host two runs of main at 32 requests differed by up to 16%, so smaller differences are ties. This branch alone was checked with the CPU suite, ruff and the GPU PD tests above.

Series. Draft PRs from one study of serving looped models. They are independent except #152, which is stacked on #147; several touch the same files, so later ones may need a rebase after earlier ones merge.

• [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU*up for CUDA graph decode #146 [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU·up for graph decode: bitwise, 1.14–1.44× on Ouro
• [KV] Pack recurrent depth planes for partially recurrent models #147 [KV] Pack recurrent depth planes: bitwise, 1.88× Huginn KV capacity
• [Scheduler] Fuse fixed-depth decode loops into one submission, keeping the per-loop prefill share #148 [Scheduler] Fused decode loops with the per-loop prefill share: per-row arithmetic unchanged
• [Engine] Packed prefill resolves automatically, for Huginn too #149 [Engine] Automatic packed prefill, Huginn included: accuracy-gated, 2.0–2.4× on 2048-token prompts
• [PD] Coalesced transfers, UCX error-handling mode, larger default budget #150 [PD] Huginn coordinator config, coalesced transfers, UCX error handling, 2 GiB default budget ← this PR
• [Perf] Host-path costs: prefix-key caching, O(1) free-block count, per-run KV bookkeeping, one argmax per batch #151 [Perf] Host-path costs in admission, KV bookkeeping and sampling
• [Huginn] Opt-in prefix caching with position-keyed recurrent-state noise (stacked on #147) #152 [Huginn] Opt-in prefix caching with position-keyed noise (stacked on [KV] Pack recurrent depth planes for partially recurrent models #147): 1.91× session throughput