Static batched decode (LLaMA + Qwen3): B independent sequences per step, up to 41x aggregate throughput#129

Open

mikepapadim wants to merge 11 commits into

beehive-lab:feat/mma_cudafrom

mikepapadim:feat/static-batched-decode

Commits on Jul 11, 2026

Add static batched decode: B independent LLaMA sequences per step, per-slot KV, 41x aggregate throughput
mikepapadim
committed
Extend static batched decode to Qwen3 (per-head QK-norm + split-half RoPE decode kernels); CUDA-backend docs + multi-model perf
mikepapadim
committed
Add continuous (iteration-level) batching: evict-on-stop + admit-from-queue, +20% throughput / +24% utilization vs static wave
mikepapadim
committed
Add PagedAttention: block-pool KV + per-slot block table, 10.7x less KV memory at ~1% overhead (LLaMA)
mikepapadim
committed
Add prefix caching: shared prompt prefix prefilled once into pinned blocks, +85% throughput / 85% fewer prefill tokens
mikepapadim
committed
Extend paging + prefix caching to Qwen3 (split-half paged RoPE; shared paged flash kernel); +104% throughput with prefix cache
mikepapadim
committed
Doc: add exact prompts + per-result engine flags for reproduction
mikepapadim
committed
Skip full-vocab logits GEMM on pure-prefill steps (logits only for sampling slots)
mikepapadim
committed
On-device greedy sampling (GPU argmax over logits): transfer B token ids not the 65MB logits tensor, +30% throughput
mikepapadim
committed
Doc: on-device sampling section + numbers
mikepapadim
committed

Commits on Jul 14, 2026

Doc: Qwen3 batched-decode quick-reproduce (copy-paste static + continuous serving) with fresh verified numbers; engine javadoc reproduce steps
mikepapadim
committed