[0z5a]

Summary

LAST_EXITED keeps one KV plane per recurrence depth and reserves every layer's slot in every plane. For a model whose prelude and coda layers run once per token (Huginn), only plane 0 ever holds those layers: of Huginn-0125's 8 layers × 32 planes = 256 layer slots per token, 124 are never written. The pool fills with dead slots, so few requests fit and throughput stops growing with concurrency (on RTX 5090 the pool held 1,328 tokens, two of our requests; on GH200 main plateaus near 65 tok/s from 32 requests on).

With CacheConfig.pack_recurrent_planes (default on) each plane maps only the layer slots it can hold: plane 0 all layers, deeper planes the recurrent ones. A page becomes a run of blocks of num_layers slots (entry = block × num_layers + slot in a flattened view of the pool), so a Huginn page needs 17 blocks instead of 32. Writes, attention, early-exit plane copies and the async finalize kernel take an entry stride and first-layer offset; PD sends each plane's live slot range (plane_slots, checked on connect) and the coordinator prices requests in blocks per page. Values never change, only their addresses. Fully recurrent models (Ouro) keep the old layout. --no-pack-recurrent-planes restores it.

Measurements

GH200, Huginn-0125 (32 loops, sync, CUDA graphs), same KV pool of 24,165 blocks, 512-token prompts, 64 output tokens, one run per level:

C	unpacked tok/s	packed tok/s	ratio	TTFT p50 unpacked → packed
1	17.5	17.3	0.99×	0.3 → 0.4 s
8	77.0	76.0	0.99×	1.3 → 1.3 s
32	100.9	114.6	1.14×	5.5 → 5.6 s
64	103.2	119.2	1.16×	15.7 → 10.9 s
128	107.8	114.5	1.06×	36.0 → 28.6 s

• Tokens in the pool: 12,080 unpacked → 22,736 packed (1.88×). 2048-token prompts: 1.13× / 1.11× at 8 / 32.
• Token identity: 233/233 requests.
• PD: 2.79 MB per prompt token sent instead of 5.41 MB (only live slots). In PD itself, packed and unpacked ran alike at 32 requests (two runs each: 91.4–92.6 vs 92.7–96.2 tok/s); the decode pool is not the limit at that load.
• Together with fused decode loops and the exact kernels (other PRs of this series), Huginn served 1.49–1.83× faster than main across 1–128 requests, 231/233 identical.

Platforms. GH200: torch 2.14.0+cu130, Triton 3.8.0, NIXL 1.4.1. RTX 5090: torch 2.13.0+cu130, Triton 3.7.1. Models: Ouro-1.4B (24 layers, 4 loops, fixed depth) and Huginn-0125 (2 prelude + 4 recurrent + 2 coda layers, 32 loops). Burst workloads issue C requests at once with random-token prompts; session workloads replay multi-turn agent traces with the prefix cache on.

Tests

• CPU suite: 836 passed; ruff passes. New CPU tests for the packed layout, routing tables, PD segments (exactly the live slots are sent, packed or not) and slot checks.
• GPU test (GH200): tiny Huginn with CUDA graphs and early-exit copies, packed vs unpacked, token for token (sync scheduling, which Huginn requires).
• GPU tests on this branch (GH200, head cc59ce9, flash-attn-3 3.0.1 and flash-attn-4 4.0.0b30 installed, pytest --run-gpu tests/test_huginn.py tests/test_kv_cache.py tests/test_async_state.py tests/test_prefix_growth.py tests/test_cdb_runtime.py): 143 passed, 1 failed: test_cuda_memory_plan_reclaims_allocator_residue_on_recreation, whose KV budget moved by 50 MiB against a 16 MiB tolerance. It fails the same way on main (66920da) on this host (46 MiB), so this branch adds no failure. Without FlashAttention installed, the FlashAttention tests failed at import on this branch and on main (66920da) alike.

Notes

The GPU numbers come from a development branch that contains all changes of this series; most cells are single runs, and on the shared GH200 host two runs of main at 32 requests differed by up to 16%, so smaller differences are ties. This branch alone was checked with the CPU suite, ruff and the GPU tests above.

Series. Draft PRs from one study of serving looped models. They are independent except #152, which is stacked on #147; several touch the same files, so later ones may need a rebase after earlier ones merge.

• [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU*up for CUDA graph decode #146 [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU·up for graph decode: bitwise, 1.14–1.44× on Ouro
• [KV] Pack recurrent depth planes for partially recurrent models #147 [KV] Pack recurrent depth planes: bitwise, 1.88× Huginn KV capacity ← this PR
• [Scheduler] Fuse fixed-depth decode loops into one submission, keeping the per-loop prefill share #148 [Scheduler] Fused decode loops with the per-loop prefill share: per-row arithmetic unchanged
• [Engine] Packed prefill resolves automatically, for Huginn too #149 [Engine] Automatic packed prefill, Huginn included: accuracy-gated, 2.0–2.4× on 2048-token prompts
• [PD] Coalesced transfers, UCX error-handling mode, larger default budget #150 [PD] Huginn coordinator config, coalesced transfers, UCX error handling, 2 GiB default budget
• [Perf] Host-path costs: prefix-key caching, O(1) free-block count, per-run KV bookkeeping, one argmax per batch #151 [Perf] Host-path costs in admission, KV bookkeeping and sampling
• [Huginn] Opt-in prefix caching with position-keyed recurrent-state noise (stacked on #147) #152 [Huginn] Opt-in prefix caching with position-keyed noise (stacked on [KV] Pack recurrent depth planes for partially recurrent models #147): 1.91× session throughput