[0z5a]

Summary

At batch 1 an Ouro-1.4B token launches about 5,700 GPU kernels, most of them small elementwise and reduction kernels from RMSNorm (8 per norm, 4 norms per layer), RoPE and SiLU·up, repeated for 24 layers × 4 loops. CUDA graphs remove their launch cost but not their minimum run time. This PR adds fused Triton kernels for the three that are bitwise identical to the PyTorch reference, and uses them only while a CUDA graph is being captured.

• RMSNorm: torch still computes the mean, so the reduction order is the reference's; one Triton kernel squares, one computes rsqrt(mean + eps), the scale, the cast and the weight multiply. 3 launches instead of 8 (4 instead of 9 with a residual, which torch adds).
• RoPE and SiLU·up: one kernel each.
• Forced rounding points (kernels/rounding.py): inline PTX cvt.rn.*.f32 / mul.rn.f32. Writing the same casts as torch is not enough: Triton/LLVM keep an fp32 product live across .to(bf16).to(f32) and contract a*b+c into an FMA. Our first fused residual epilogue computed round(r + w*n) where torch computes round(r + round(w*n)); 18.4% of 75,776 bf16 elements differed by one ulp.
• Graph scope (VLLM_RLT_FUSED_SCOPE=graph, default): a Triton launch from Python costs more host time than the torch ops it replaces, so eager paths (prefill, sync eager decode) stay on torch. With fusion in every scope, sync eager decode on GH200 ran at 0.40× and prefill steps took 33–46% longer.
• VLLM_RLT_FUSED_NORM / _ROPE / _ACT = exact (default) | approx | 0. The approximate one-kernel versions stay opt-in: faster, but they change greedy outputs.

Measurements

GH200, Ouro-1.4B, 512-token prompts, 128 output tokens, async scheduling + CUDA graphs, all three arms in one job, one run per level:

	C=1	C=8	C=32	C=128
unfused tok/s	61.1	362.5	711.5	666.6
exact set	1.44×	1.44×	1.23×	1.14×
approximate set	1.70×	1.67×	1.42×	1.39×

• Kernels per token at batch 1: 5689 → 2885 (exact) → 1917 (approximate); GPU time per token 13.05 → 8.91 → 7.63 ms.
• Token identity vs unfused: exact 169/169 requests; approximate 155/169.
• GSM8K, 87-question subset, greedy, serial (repository harness): exact 62/87 vs unfused 62/87 on GH200, 0 sequences differ. On RTX 5090 the approximate RMSNorm alone scored −3.45 points (paired s.e. 2.56) and the full approximate set −2.30 (s.e. 1.62); both fail the 1-point gate, which is why they stay opt-in.
• SiLU·up exact: 0 mismatching elements in 8.3e8 (bf16/fp16/fp32 × three gate scales).

Platforms. GH200: torch 2.14.0+cu130, Triton 3.8.0, NIXL 1.4.1. RTX 5090: torch 2.13.0+cu130, Triton 3.7.1. Models: Ouro-1.4B (24 layers, 4 loops, fixed depth) and Huginn-0125 (2 prelude + 4 recurrent + 2 coda layers, 32 loops). Burst workloads issue C requests at once with random-token prompts; session workloads replay multi-turn agent traces with the prefix cache on.

Tests

• CPU suite: 829 passed (kernels skip without CUDA); ruff check and format pass.
• GPU tests (GH200): RMSNorm exact bitwise with and without residual (bf16/fp16/fp32 × three widths), RoPE exact bitwise, SiLU exact; approximate kernels checked against single-rounding oracles.
• GPU tests on this branch (GH200, head e8cd884, flash-attn-3 3.0.1 and flash-attn-4 4.0.0b30 installed, pytest --run-gpu tests/test_normalization.py tests/test_rotary_embedding.py tests/test_ouro.py tests/test_cuda_graph.py): 124 passed, 2 skipped (fused RoPE needs a power-of-two half dimension). Without FlashAttention installed, the FlashAttention tests failed at import on this branch and on main (66920da) alike.

Notes

The GPU numbers come from a development branch that contains all changes of this series; most cells are single runs, and on the shared GH200 host two runs of main at 32 requests differed by up to 16%, so smaller differences are ties. This branch alone was checked with the CPU suite, ruff and the GPU tests above.

Series. Draft PRs from one study of serving looped models. They are independent except #152, which is stacked on #147; several touch the same files, so later ones may need a rebase after earlier ones merge.

• [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU*up for CUDA graph decode #146 [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU·up for graph decode: bitwise, 1.14–1.44× on Ouro ← this PR
• [KV] Pack recurrent depth planes for partially recurrent models #147 [KV] Pack recurrent depth planes: bitwise, 1.88× Huginn KV capacity
• [Scheduler] Fuse fixed-depth decode loops into one submission, keeping the per-loop prefill share #148 [Scheduler] Fused decode loops with the per-loop prefill share: per-row arithmetic unchanged
• [Engine] Packed prefill resolves automatically, for Huginn too #149 [Engine] Automatic packed prefill, Huginn included: accuracy-gated, 2.0–2.4× on 2048-token prompts
• [PD] Coalesced transfers, UCX error-handling mode, larger default budget #150 [PD] Huginn coordinator config, coalesced transfers, UCX error handling, 2 GiB default budget
• [Perf] Host-path costs: prefix-key caching, O(1) free-block count, per-run KV bookkeeping, one argmax per batch #151 [Perf] Host-path costs in admission, KV bookkeeping and sampling
• [Huginn] Opt-in prefix caching with position-keyed recurrent-state noise (stacked on #147) #152 [Huginn] Opt-in prefix caching with position-keyed noise (stacked on [KV] Pack recurrent depth planes for partially recurrent models #147): 1.91× session throughput