[Benchmark] NVFP4, RTX 5070 + RTX 5060 Ti: stage trim, arena pinning and a Cyrillic draft vocabulary take decode from 46.2 to 67.0 tok/s #4261

Benchmark report, no engine change. One PC, two mismatched Blackwell cards, NVFP4 GPTQ of Qwen3.8-Flash-Next at 262,144 tokens. Measured on 2026-10-26 with a self-built engine 0.1.51 (CMAKE_CUDA_ARCHITECTURES=120).

Hardware: RTX 5070 12 GB + RTX 5060 Ti 16 GB, both sm_120, both on PCIe 5.0 x8 (engine probe 28.7–28.9 GB/s), neither drives a display. Ryzen 9 9950X3D2, 16 cores / 32 threads, AVX-512, one socket. 125.5 GiB RAM, page file 31.2 + 62.5 GiB, commit limit 219.3 GiB. Windows 11, driver 580.65+, CUDA 13.0.

Model: Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata, 119.17 GiB; its own pack/; bundled expert-profile.bin (24,576 ranked pairs); PLE table ple-fp8.gguf (47.68 GiB). Context 262,144, --kv int8, no KV streaming, expert arena in RAM (neither --resident-experts nor --mmap-experts), --ple-io direct, --expert-cache auto, --prefill auto, --spec 4 --spec-min-p 0.5, --host-core sibling, --vram-reserve-mib 0. Cards: "gpu": [1, 0] — 5060 Ti holds layers 0–27, 5070 holds 28–47 plus the output head and the draft layer.

Result: decode 46.2 → 67.0 tok/s, prompt 1,567.7 → 2,161.4 tok/s. Three opt-in settings, each tested alone, 6 runs per point with the warm-up discarded:

1. STRATA_STAGE_TRIM=1 on NVFP4

Opt-in upstream and, going by the community folder list, unmeasured on NVFP4. It gave the second card back 2.54 GiB of VRAM:

The point for other layer-split users: on a split the prompt chunk of every card is clamped to what the smallest cache can lend, so the small cache on the 12 GB card was capping the prompt speed of both cards. The engine says it itself — WARNING: prompt chunk 5376, not 8192: CUDA1 has 1128 expert-cache slots, and a 8192-token chunk borrows 1128 of them. Removing that clamp is where the prompt gain comes from.

Split-point sweep with trim on (K = layers on the first card):

K=28 was measured twice in two separate engine sessions (54.2, 53.5).

2. A trap with auto under trim

With trim on, layer_split: auto picks K=24 and decodes at 49.4, while the hand-set K=28 gives 53.9 — an 8% loss. The search prices decode without the new slot distribution and warns about it itself (the prediction may not rank them; it prices decode, not the prompt). Worth knowing before anyone adopts trim and then trusts auto to re-place the split.

3. The shipped draft vocabulary does not cover Cyrillic

draft_vocab.bin as shipped is a CJK subset: of the model's 18,580 Cyrillic tokens it holds 142 (0.76%). On Russian text the drafter can therefore propose almost nothing.

The Cyrillic subset is built from the English base plus the script, so it is smaller than the CJK one — the draft head gives back 123 MiB. English was measured separately at 67.3 tok/s (65.4–71.3, 5 runs): no loss, because the subset is the English base plus a script rather than a replacement. Note the file is shared by every model in the data folder and is replaced by a plain copy — there is no config key for it, and re-running model setup overwrites it.

Limitations, stated plainly

Numbers come from my own fixed prompt and scripts, not from strata bench run, so they are not standard-protocol results and should not be compared row-for-row with them. One Russian prompt (max_tokens 700, temperature 1.0, top_p 0.95, top_k 20) plus one English prompt for the language-cost check, one sampling setting. Prompt figures are from the engine line prompt N tokens = 0 reused + N read in … ms (… tok/s) on unique text (a Guid in every paragraph, so reused = 0), one run per point. The model was fully unloaded and reloaded between configurations (layer_split and env are read only at engine start).

No TTFT, no needle/recall, no coding tasks with test runs, no tool calls, no concurrency, no power capture.

Spread inside one configuration (4–9 tok/s) is wider than the gap between neighbouring split points, so only the peak position and the large steps are meaningful.

Text is not bit-identical to the untrimmed run: with fewer experts in VRAM more rows are computed by the CPU pool, which rounds differently. Checked for encoding damage only (0 U+FFFD, 0 control characters in a 3,807-character answer), not for answer quality.

Loading takes ~90 s instead of ~75 with trim on, because the later stages load their weights after the split search (late_weights).

Negative results, each rejected for a stated reason

--pipeline-windows 2 (auto-off on CUDA; the fork's reported win came with its own re-tuned split search — upstream measured +5.4% on code against 2.8% noise and −13.2% with STRATA_MTP_STAGE0=1), --adapt-async 1 (not with a layer split in the arena mode), STRATA_PIN_BY_NEED=1 (not with a layer split or a peer GPU), STRATA_EXCHANGE_ROTATE=1 (page-lock of the 58 GiB complement refused on Windows), --kv fp16 (real fp16, but halves the slots 4,294 → 3,072 and costs 12.6% decode), --ple-io ram (47.68 GiB of RAM for no gain: 40.1 vs 40.2), --mmap-experts (63.5 vs 67.7–73.3 in the arena), --resident-budget-gib N (reads the remainder from disk by definition; 63/70/100 give the same figure), --expert-cache-per-layer (−21% decode).