[0z5a]

Summary

Four host-side fixes found by profiling multi-turn session workloads (cProfile inside the benchmark; in one cell the GPU was 13% busy while the Python process was at 100%):

• Scheduler._plan_admission rehashed every waiting prompt (one chained SHA-256 per block) on every admission attempt, and num_free_blocks rescanned every prefix entry on each capacity check. Prompt keys are now computed once per request, and the manager keeps the count of cache-only blocks (refs == 1) current at every reference-count change, so num_free_blocks is O(1).
• Growth reservations are computed once per admission tick instead of once per candidate.
• Rows of a host KV batch are grouped once into runs of consecutive positions per (allocation, depth); per-layer write/attend bookkeeping advances one written record per run (add_run, identical to sequential add) and checks the strongest prefix requirement per run. A 2048-token prefill chunk costs one call per layer instead of 2048. The per-layer written contract is unchanged. In the session profile this bookkeeping was 31M calls, about 20 s of 146 s.
• An all-greedy CODA batch samples with one argmax launch (Sampler.sample_batch) instead of one per row.

Measurements

RTX 5090, Ouro-1.4B, 32 agent sessions, 6 turns, prefix cache on, one run per arm (prefix-key and free-block changes vs main):

	tok/s	TTFT p50	Python calls
per-row prefill	+2.6%	8.4 → 7.1 s	−19%
packed prefill	+4.7%	2.57 → 1.80 s	—

The other two changes were not measured on their own.

Platforms. GH200: torch 2.14.0+cu130, Triton 3.8.0, NIXL 1.4.1. RTX 5090: torch 2.13.0+cu130, Triton 3.7.1. Models: Ouro-1.4B (24 layers, 4 loops, fixed depth) and Huginn-0125 (2 prelude + 4 recurrent + 2 coda layers, 32 loops). Burst workloads issue C requests at once with random-token prompts; session workloads replay multi-turn agent traces with the prefix cache on.

Tests

• CPU suite: 831 passed; ruff passes. Randomized tests check the free-block counter against the old scan after each allocate/free/grow/publish, and add_run against sequential add; a sampler test covers the batched argmax.

Notes

The GPU numbers come from a development branch that contains all changes of this series; most cells are single runs, and on the shared GH200 host two runs of main at 32 requests differed by up to 16%, so smaller differences are ties. This branch alone was checked with the CPU suite and ruff.

Series. Draft PRs from one study of serving looped models. They are independent except #152, which is stacked on #147; several touch the same files, so later ones may need a rebase after earlier ones merge.

• [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU*up for CUDA graph decode #146 [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU·up for graph decode: bitwise, 1.14–1.44× on Ouro
• [KV] Pack recurrent depth planes for partially recurrent models #147 [KV] Pack recurrent depth planes: bitwise, 1.88× Huginn KV capacity
• [Scheduler] Fuse fixed-depth decode loops into one submission, keeping the per-loop prefill share #148 [Scheduler] Fused decode loops with the per-loop prefill share: per-row arithmetic unchanged
• [Engine] Packed prefill resolves automatically, for Huginn too #149 [Engine] Automatic packed prefill, Huginn included: accuracy-gated, 2.0–2.4× on 2048-token prompts
• [PD] Coalesced transfers, UCX error-handling mode, larger default budget #150 [PD] Huginn coordinator config, coalesced transfers, UCX error handling, 2 GiB default budget
• [Perf] Host-path costs: prefix-key caching, O(1) free-block count, per-run KV bookkeeping, one argmax per batch #151 [Perf] Host-path costs in admission, KV bookkeeping and sampling ← this PR
• [Huginn] Opt-in prefix caching with position-keyed recurrent-state noise (stacked on #147) #152 [Huginn] Opt-in prefix caching with position-keyed noise (stacked on [KV] Pack recurrent depth planes for partially recurrent models #147): 1.91× session throughput