[0z5a]

Summary

Stacked on #147 (it uses the per-plane layer map); review only the top commit.

Huginn could never reuse a prefix. publish_prefix required a contiguous written prefix in every layer of every plane, and Huginn's prelude/coda layers write plane 0 only, so no Huginn page was ever published. Coverage is now checked per plane over the layers it holds (all layers at depth 0, the recurrent layers deeper); fully recurrent models are unchanged.

A cache hit still changed outputs. Huginn draws each token's initial latent state from a per-request RNG stream in processing order, so a request whose first k prompt tokens come from the cache draws k fewer values, and every later token gets different noise than in a cold run (the prefill chunk size changes the draws the same way). With VLLM_RLT_HUGINN_POSITION_NOISE=1 the noise is a function of (request seed, position, channel): a stage-wise SplitMix64-style hash gives a 24-bit uniform, and an inverse normal CDF maps it to the same truncated normal (std = initializer_range, ±3σ). 64-bit wrap-around is identical on CPU and CUDA. It keeps the distribution but changes which samples a seeded run sees, so it is opt-in, together with VLLM_RLT_HUGINN_PREFIX_CACHE=1 for the engine guard.

Measurements

GH200, Huginn-0125 (32 loops, graphs), 16 agent sessions, at most 4 turns, prompts up to 2000 tokens, 59 requests, one run; measured with plane packing, packed prefill and fused decode loops, position noise on in both arms:

	tok/s	TTFT p50	TTFT p90	hit rate
prefix cache off	13.8	1.93 s	6.00 s	0%
prefix cache on	26.3 (1.91×)	0.44 s	1.90 s	78.1%
main (no packing, per-row prefill, no cache)	12.4	5.83 s	11.76 s	—

Cached vs cold on the GPU (GH200, Huginn-0125, 32 loops, CUDA graphs, packed prefill, eight 512-token prompts run one at a time, 48 greedy tokens; the second run reuses 496 prompt tokens from the cache):

	full prefix hit = cold	longer prompt, partial hit = cold
position-keyed noise	6/8	8/8
sequential noise (upstream)	6/8	8/8

Position keys removed the one divergence caused by the noise order (a prompt that changed at its first token with the sequential generator matched with position keys). Two divergences remain and come from prefill arithmetic: a cached run prefills only the suffix, so its GEMMs see fewer rows, and one of the two prompts also changed when only the prefill chunk size changed (15/16 cold runs matched between chunks of 2048 and 128 tokens). On the GPU, reuse therefore behaves like a change in batch composition on top of the change of samples. The noise itself matters at 32 loops: the two generators gave the same tokens for 11/16 cold prompts.

Platforms. GH200: torch 2.14.0+cu130, Triton 3.8.0, NIXL 1.4.1. RTX 5090: torch 2.13.0+cu130, Triton 3.7.1. Models: Ouro-1.4B (24 layers, 4 loops, fixed depth) and Huginn-0125 (2 prelude + 4 recurrent + 2 coda layers, 32 loops). Burst workloads issue C requests at once with random-token prompts; session workloads replay multi-turn agent traces with the prefix cache on.

Tests

• CPU suite: 839 passed; ruff passes. New tests: cached and cold runs, and prefill chunks of 2 and 8 tokens, give identical tokens with position noise; the noise is batch-independent, seed- and position-keyed, free of seed/channel aliasing, and has the truncated-normal moments and tail mass; publication covers only the layers each plane holds.

Notes

The GPU numbers come from a development branch that contains all changes of this series; most cells are single runs, and on the shared GH200 host two runs of main at 32 requests differed by up to 16%, so smaller differences are ties. This branch alone was checked with the CPU suite and ruff.

Series. Draft PRs from one study of serving looped models. They are independent except #152, which is stacked on #147; several touch the same files, so later ones may need a rebase after earlier ones merge.

• [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU*up for CUDA graph decode #146 [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU·up for graph decode: bitwise, 1.14–1.44× on Ouro
• [KV] Pack recurrent depth planes for partially recurrent models #147 [KV] Pack recurrent depth planes: bitwise, 1.88× Huginn KV capacity
• [Scheduler] Fuse fixed-depth decode loops into one submission, keeping the per-loop prefill share #148 [Scheduler] Fused decode loops with the per-loop prefill share: per-row arithmetic unchanged
• [Engine] Packed prefill resolves automatically, for Huginn too #149 [Engine] Automatic packed prefill, Huginn included: accuracy-gated, 2.0–2.4× on 2048-token prompts
• [PD] Coalesced transfers, UCX error-handling mode, larger default budget #150 [PD] Huginn coordinator config, coalesced transfers, UCX error handling, 2 GiB default budget
• [Perf] Host-path costs: prefix-key caching, O(1) free-block count, per-run KV bookkeeping, one argmax per batch #151 [Perf] Host-path costs in admission, KV bookkeeping and sampling
• [Huginn] Opt-in prefix caching with position-keyed recurrent-state noise (stacked on #147) #152 [Huginn] Opt-in prefix caching with position-keyed noise (stacked on [KV] Pack recurrent depth planes for partially recurrent models #147): 1.91× session throughput ← this PR