🚧 [WIP] MoE cache, CPU experts and GGUF support #671

Tracks the refactor of the MoE expert cache and the CPU expert path and GGUF support.

Where it stands

• The GPU slot cache splits into slot pools, one per layer or one per quant (feat(moe): split the offload expert cache into slot pools #689), and each MoE layer has its own expert method and bank set (feat(moe): serve experts of several formats through per-format slot pools #700).
• The engine takes each MoE layer's own quant method, and the CPU expert executor serves a weight format per layer (feat(moe): serve experts of several formats through per-format slot pools #700).
• Several decisions made before the model is built still read one model-wide ModelConfig.expert_quant: whether fused is allowed, --moe-cpu-layers auto, the automatic offload / hybrid pick, and some family weight readers.
• GGUF checkpoints load through the quant layer and run on vendored ggml-cuda kernels (feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels #690), TP 1, for the families under Model Support. Qwen3.5 / 3.6 / 3.8, Qwen3-VL, Gemma-4, GLM-5.3-Flash, DeepSeek-V4-Flash-Vision-Exp, MiniMax-M3 and Muse-Glimmer also take image input from an mmproj (feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj #803, feat(gguf): serve image input on glm5-next ggufs from a glm5v mmproj #810, feat(gguf): serve image input on deepseek4 ggufs from a deepseek4v mmproj #816, feat(gguf): serve image input on qwen3vl ggufs from a deepstack qwen3vl_merger mmproj #827, feat(gguf): serve image input on minimax-m3 ggufs from a minimax_m3 mmproj #829, feat(gguf): serve image input on muse-glimmer ggufs from a muse-glimmer mmproj #830, see GGUF vision towers (mmproj): survey and plan #794); the other families serve text only. cpu runs every GGUF expert type through ggml-cpu (feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu #704); hybrid above 8 tokens per step is limited by the first known issue below.

Work items

Cache

• Slot pools: one pool per layer or one per quant (quant tag and bank layout), sized equally or from an expert-use profile. feat(moe): split the offload expert cache into slot pools #689 @jason-fxz
• Bank sets that differ per layer: the banks are the union of the formats' roles, a layer without a role gets a zero-byte placeholder, and each format gets its own slot pool. feat(moe): serve experts of several formats through per-format slot pools #700 @andy-yang-1 @jason-fxz
• Pin-capped hosts: under hybrid, OS-locked layers keep slot pools of their own and fetch part of their misses through a pinned staging buffer, so models with several slot pools (mixed-type GGUF) combine with --moe-cpu-layers; --moe-cpu-layers auto finds the pin limit by pinning until the driver refuses. feat(moe): fetch locked-layer experts through pinned staging #742 @jason-fxz

Mixed quant per layer

• Engine, loaders and FTW checkpoints take each layer's own expert method; the CPU executor gets every layer's format as (name, options) and treats all formats alike. feat(moe): serve experts of several formats through per-format slot pools #700 @andy-yang-1 @jason-fxz
• Resident (fused) experts of several formats.
• Record each layer's expert quant in ModelConfig (keep expert_quant as the most common one for compatibility) and make the decisions taken before the model is built per layer: fused eligibility, --moe-cpu-layers auto, the automatic offload / hybrid pick, and the family weight readers (qwen3_5_moe reads stacked bf16 experts only when expert_quant == "none").
• ft bench bw: record one entry per layer kind, keyed by (format, (H, I), options), e.g. ("bf16", (H, I)), ("nvfp4", (H, I)), ("gguf", (H, I), Q4_0, Q6_K), and let the engine pick hybrid and the fetch fraction per layer. (feat(moe): bench hybrid fetch fractions per moe layer kind #734)

CPU expert executor

• Split the executor from the weight formats and the GPU handshake (refactor(cpu-moe): split the executor into weight formats and gpu sync backends #670). @jason-fxz
• Kernel IO for the handshake: a publish kernel writes a step's inputs into the pinned IO block and rings the doorbell, or skips a step without CPU routes; a combine kernel reads the CPU rows back. The coordinator skips tasks without routed experts (refactor(cpu-moe): split the executor into weight formats and gpu sync backends #670). @jason-fxz
• Hybrid decode: split the routes between GPU and CPU inside the hybrid ensure kernel instead of torch ops (perf(moe): split hybrid decode routes inside the ensure kernel #685). @jason-fxz
• Batch size > 1: split the work by expert, so an expert shared by several tokens is read once.

GGUF

Kernels and loading

• GPU: llama.cpp's ggml-cuda MMQ / MMVQ kernels for every GGUF type that has an MMQ kernel, prefill and decode (feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels #690). @jason-fxz
• CPU: llama.cpp's ggml-cpu row kernels (quantized, and f32 / f16 / bf16), vendored unchanged and built for AVX2 as static libraries linked into _cpu_moe; one gguf executor format per layer's (gate_up type, down type), and every type with a ggml-cpu vec_dot works. Replaces the q4_0 CPU format. feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu #704 (on top of feat(moe): serve experts of several formats through per-format slot pools #700) @jason-fxz
• Tokenizer: build GGUF tokenizers on transformers 5.18, which keeps USER_DEFINED and CONTROL tokens such as <think> whole, with llama.cpp's stop tokens (build(deps): upgrade transformers to 5.18 #698). @jason-fxz

Model Support

Every model family FreeToken registers; the backticked name is the GGUF architecture llama.cpp's converter writes.

• Qwen3.8-Flash-Next (qwen4exp) (feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels #690). @jason-fxz
• Qwen3.5 / Qwen3.6, also Qwen3.8-27B (qwen35, qwen35moe). (feat(gguf): serve Qwen3.5 dense and MoE GGUF checkpoints #701)
• Gemma-4 26B-A4B (gemma4) (feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels #690). @jason-fxz
• Dense Gemma-4 (gemma4). (feat(gguf): serve Gemma 4 dense GGUF checkpoints #706)
• E-series Gemma-4 (gemma4)
• Llama (llama). (feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints #722)
• Qwen2 (qwen2). (feat(gguf): serve Qwen2, Qwen3 and Qwen3 MoE GGUF checkpoints #707)
• Qwen3 / Qwen3-MoE (qwen3, qwen3moe). (feat(gguf): serve Qwen2, Qwen3 and Qwen3 MoE GGUF checkpoints #707)
• Qwen3-VL / Qwen3-VL-MoE (qwen3vl, qwen3vlmoe). (feat(gguf): serve qwen3vl and qwen3vlmoe gguf checkpoints text only #807)
• MiniMax-M2 (minimax-m2). (feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints #725)
• MiniMax-M3 (minimax-m3). (feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints #725)
• DeepSeek-V4 (deepseek4). (feat(gguf): serve DeepSeek-V4 Flash and Pro GGUF checkpoints on deepseek4 #774)
• DeepSeek-V4.1 (llama.cpp has no converter for it).
• Muse-Glimmer (muse-glimmer). (feat(gguf): serve Muse-Glimmer GGUF checkpoints #724)
• Mistral models converted before llama.cpp had mistral3 (llama): Mistral-7B, Mistral-Nemo, Mistral-Small-3.1 / 3.2, Magistral, Devstral-2507. (feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints #722)
• Mistral 3 (mistral3).
• gpt-oss (gpt-oss).
• GLM-4.5 / 4.7 (glm4moe). (feat(gguf): serve GLM-4.5, GLM-4.6 and GLM-4.7 GGUF checkpoints #731)
• GLM-5 / 5.1 / 5.2 (glm-dsa).
• GLM-5.3-Flash (glm5-next). (feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next #721)
• Flint (flint). (feat(flint): add nvfp4 modelopt and gguf support #797)

Vision towers

• Image input from llama.cpp mmproj files, tracked in GGUF vision towers (mmproj): survey and plan #794: Qwen3.5 / 3.6 / 3.8 and Gemma-4 (feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj #803), GLM-5.3-Flash (feat(gguf): serve image input on glm5-next ggufs from a glm5v mmproj #810), DeepSeek-V4-Flash-Vision-Exp (feat(gguf): serve image input on deepseek4 ggufs from a deepseek4v mmproj #816), Qwen3-VL (feat(gguf): serve image input on qwen3vl ggufs from a deepstack qwen3vl_merger mmproj #827), MiniMax-M3 (feat(gguf): serve image input on minimax-m3 ggufs from a minimax_m3 mmproj #829), Muse-Glimmer (feat(gguf): serve image input on muse-glimmer ggufs from a muse-glimmer mmproj #830, images only).

Known issues

• MoE MMQ (the GPU path above 8 tokens) hits an illegal memory access when one token routes to the same expert slot more than once. hybrid does this on every step that has CPU routes: they are parked on slot 0 with weight 0 because the GGUF MoE kernel cannot skip inactive slots. So hybrid crashes above 8 tokens per step for any quantized expert type; Q4_0 is the only one it runs today. Open fix: fix(moe): skip inactive gguf routes in hybrid decode #823.
• Qwen3.8-Flash-Next with --ple-backend disk (wait-sync): the first CUDA graph replay of a batch size can hang, because the graph waits on a host flag that the engine sets only after replay() returns and the first replay also uploads the graph. Seen at 16 concurrent requests on the GGUF; not GGUF-specific. fix(engine): upload cuda graphs at capture so a stream wait cannot block the first launch #697 @jason-fxz
• MMQ / MMVQ use 32-bit block offsets, so a weight view of more than 2^31 blocks is not supported, e.g. a Qwen3.8-Flash-Next offload pool of Q4_0 / Q8_0 experts above 20,971 slots. fix(gguf): reject ggml weights past the 32-bit block offsets of MMQ / MMVQ #699 makes it raise instead of faulting. @jason-fxz
• IQ1_M experts have no MMQ kernel, so MoE steps above 8 tokens fail. feat(kernel): add an iq1_m mmq kernel for moe steps above 8 tokens #835
• NVFP4 GGUF .scale: llama.cpp's converter keeps each NVFP4 tensor's weight_scale_2 in a separate <name>.scale tensor. It is now applied on the GPU and CPU paths, and the gemma4, llama, qwen3, qwen3moe, qwen35 and qwen35moe loaders read it (feat(gguf): apply the weight scales of nvfp4 gguf tensors #732). An NVFP4 token_embd is read since feat(gguf): read NVFP4 token_embd rows in the vendored get_rows #801; like llama.cpp, which cannot load such a file, one with a .scale is refused.
• NVFP4 GGUF .scale, still refused on glm4moe, qwen2, minimax-m2, minimax-m3, muse-glimmer and qwen4exp, which have no tested NVFP4 GGUF. llama.cpp's converter cannot produce a valid one for minimax-m2 and muse-glimmer.

Performance

do after basic GGUF CUDA/CPU kernel support.

• ggml CPU kernels for GGUF experts (measured on a Xeon 8480+, 12 threads): x86 quantize_row_q8_K is scalar (about 5 us per row at K=2560); IQ types are compute-bound (30-40 GB/s per expert vs about 140 GB/s for Q4_K); with several tokens on the same experts, Q4_0 / Q4_K / Q8_0 run slower than on distinct experts (cause not investigated).
• MoE MMQ: tune the tile width J of the flat grid; the automatic choice was up to 23% slower than the best J in a sweep.
• Native FP4 MMA (off unless GGML_CUDA_FP4_MMA; needs an sm_120a / sm_120f build).