Gemma 4 on the batched-decode branch: merge #120 + perf findings (CUDA graphs +13.6%)

[mikepapadim]

Gemma 4 on the batched-decode branch

Merges Gemma 4 support (PR beehive-lab#120, gemma4-new) onto the vLLM-style batched-decode work
(feat/static-batched-decode, PR beehive-lab#129) and evaluates squeezing Gemma 4 decode perf. Full
write-up: GEMMA4_BATCHED.md. Model: unsloth/gemma-4-E2B-it-Q8_0 (5 GB), RTX 4090, CUDA backend.

Results

• ✅ Clean merge — Gemma 4 coexists with all batched-decode features; builds + runs.

• ✅ Stock Gemma 4 GPU decode works — "The capital of France is **Paris**."

• ✅ CUDA graphs (model-agnostic) squeeze:

  Gemma-4-E2B Q8_0	tok/s	speedup
  no CUDA graphs	22.3 / 11.5 (200-tok)	1.0×
  CUDA graphs	25.3 / 12.3	+13.6% / +6.5%

Why full batched decode is a larger follow-up

Gemma 4 (Gemma-3n-class MatFormer) breaks the batched-decode engine's uniform-layer assumptions:
per-layer head/FFN dims, alternating sliding-window / full attention, shared-KV layers,
per-layer embeddings (PLE) + AltUp, pre+post norms + per-head Q/K RMSNorm + query scaling,
GeGLU (not SiLU), NEOX RoPE with two thetas, final logit softcap, and BF16/Q8_0
weights (no plain FP16 for the FP16 MMA path). A Gemma 4 batched-decode layer graph therefore
needs new windowed decode attention, GeGLU FFN, and NEOX RoPE kernels + Gemma
norm/scale tasks + shared-KV addressing + a PLE embedding path + a BF16-or-FP16 weight story —
a dedicated multi-file build, not a small fork.

What transfers for free

The engine-level features (on-device sampling, continuous batching, PagedAttention,
prefix caching, logits-skip) are model-agnostic and drop in once a Gemma 4 batched
forward exists. On-device sampling would help Gemma 4 more (vocab ~262 k → bigger per-step D2H);
greedy stays valid through the monotonic tanh softcap.

Tracked in the vLLM roadmap issue beehive-lab#130. Base branch: feat/static-batched-decode (PR beehive-lab#129).