[Benchmark] 2x RTX 4090 (PCIe 4.0 x16, TRX40): UD-Q4_K_XL at 307K - tuned-config metrics plus a Gen4/Gen3 x chunk/layer-major matrix #4573

TL;DR

Results only, measured 2026-10-28/29 (JST) on one machine. Unsloth UD-Q4_K_XL on 2x RTX 4090 at 307,200 configured context, engine 0.1.52, Terminal 0.2.7.

• Production steady state (chunk-major prefill, expert cache hot): decode 122-127 tok/s at 110-120K context (burst to ~240 on 82%-acceptance draft windows), cold 185,013-token read at 3,239.7 tok/s, expert-cache hit 97.6-98.0%, prompt reuse 99.6-99.9%, STALL reports 0 in 24 h.
• The new part: a single-variable link matrix. Only the BIOS per-slot PCIe speed changed (Gen4 x16 vs Gen3 x16, same machine, same config, same salted prompts): layer-major (walk) prefill is +16.1% over chunk-major at 150K on Gen4 but net -1.7% on Gen3; decode is unchanged by either link or path.
• How we made the walk engage at all (first time on this box after 47 engine starts with zero engage lines): --vram-reserve-later-mib 972 -> 1500 - the last card's reserve, not the global margin. Full audit and the correction this produced: [Engine] Layer-major prefill on a CUDA layer split: 32K prompts 2.0x faster, 0.47x drive reads (Qwen3.8-Flash-Next, v0.2.4.2) #3976 (our comments 6473012700, edited, and 6484683205).

Setup

Production steady-state metrics (2026-10-28 08:00-22:30 JST window + 10-29 morning)

Link matrix (the actual benchmark here)

Method: fresh prompts (a GUID-text corpus plus a per-request salt, so 0 reused every time) at ~32K/66K/150K nominal tokens (actual usage 33,1xx / 66,5xx / 150,9xx), max_tokens=32, temperature 0, two iterations per cell; reported rate = the engine log line prompt N tokens = 0 reused + N read in X ms (Y tok/s). Values are the mean of the two iterations. (Our earlier #3976 table used the lower of the pair for the Gen3 rows; differences are <= 25 tok/s and the conclusions are identical.)

Arm order in one morning: Gen3 chunk 08:54, Gen3 walk 09:13, Gen4 walk 09:27, Gen4 chunk 09:34 (BIOS restored between arms; pcie.link.gen.current verified 4/P2/99% util under load). Read-only observations: chunk-major itself loses ~16-22% going Gen4 to Gen3; decode never moved (124.7-129.5); walk engage confirmed by layer-major (layers 0..22) + (layers 22..48) lines in the walk arms and no VRAM for a super-chunk (734 MiB free) declines in the chunk arms.

Tuning A/B notes behind the production config (what helped, what did not)

1. STRATA_ROUTE_PRIOR=1.0 - the biggest single win. Hints the next turn's expert placement from the previous turn's routing. At the default 0 the box streamed over PCIe constantly (hit 86.7%); A/B at identical load 0 -> 0.5 -> 1.0: expert-cache hit 86.7% -> 97.6% (65 min / 196 requests window), decode median 127 tok/s (peak ~240 on full-accept draft windows), and no visible quality difference comparing output at the 4,995-token position (branch-free conversation). Downsides: placement can lag right after a hard context switch (but the hit floor stays at the old value, so it cannot break below baseline); the arena fills with the previous conversation's experts, so alternating between conversations dilutes the win (observed 88-97% band).

2. STRATA_PREFILL_LM_MARGIN_MIB=96 - a knob I believed in that turned out to be a no-op here (corrected 2026-10-29). I first credited it with making the 185K cold read run at 3,239.7 tok/s. The log audit says otherwise: the walk never engaged in 47 engine starts - the primary evaluator passes at margin 96, but the last stage declined every prompt (no VRAM for a super-chunk (734 MiB free), floor ~1.1 GiB), so every prompt ran chunk-major and 3,239.7 is chunk-major. What actually engaged the walk was the last card's reserve, --vram-reserve-later-mib 972 -> 1500 (matrix above; audit in #3976 comments 6473012700, edited, and 6484683205). Production stays at later 972 - walk deliberately off until the engagement rule lands.

3. --conversation-cache-mib 12288 --conversation-cache-slots 4 - the agent-workflow fix. Budget for parked K/V snapshots. Prompt reuse 99.6-99.9%: 403 of 114,157 tokens read (1.8 s), 115 of 115,452 (1.1 s), and it survives an engine restart (re-parking finished inside the timeout). Downsides: a parked conversation costs GB-class (about 6.5 GiB at 30K tokens) so 12 GiB is tight; the 5th conversation starts evicting, and under extreme memory pressure there is a path where re-parking an evicted conversation is refused (Windows-side evidence added to the #2253 thread).

4. --kv-resident 32768 + --kv int8. KV-stream VRAM hit 96.7-97.0%; a 115K conversation reads only ~1.7-2.7 GiB from RAM. Downside: residency eats GPU free (steady ~1,220 / ~1,143 MiB here); pushing it further shows up first as #2065-style STALL reports - tune it together with the two --vram-reserve flags.

5. --spec 4 --pipeline-windows 2 --mtp rt --spec-min-p 0.70. Pipeline speculation across the two cards (card 2 drafts the next window while card 1 verifies). Decode 121.9-126.5 tok/s at 115K context; window acceptance 48/67, 41/42, 72/101. Downsides: draft-head VRAM and pipeline-specific edge cases - a draft can cross the page table within 7 tokens of the context ceiling (reported as #4413, never hit in practice); --adapt-async 1 is off by design under layer-split + arena (blocking tier).

6. STRATA_HC_Q8=1 / STRATA_PIN_BY_NEED=1 / STRATA_ARENA_PIN_GIB=24. HC_Q8 reads the hyper-connection projections as Q8 - quality side; under layer split the bf16 copy is not evacuated either (#2427 open), so it is a judgement call of about 0.63 GiB wasted per card vs quality (took quality). PIN_BY_NEED + ARENA_PIN_GIB VirtualLocks exactly the arena bytes streaming needs (24 GiB), so OS pressure cannot shred the streaming rate; the only downside is simply less pageable RAM.

What this adds (link to #3976's rule discussion)

• @coryjett's finding (walk at 2.1x SLOWER than chunk on his 2x3090 Gen3 pair, -49%) does not generalize as "Gen3 breaks the walk": on a single-socket pair the Gen3 delta is -1.7% at 150K. The gate needs the measured link and probably the route (his pair is one card per socket), not just a generation number.
• The engine already probes each card's link at startup and distinguishes the generations (26.4 vs 13.2), though it twice read ~4x low (6.7 on a Gen4 card) when the sample caught the idle Gen1 negotiation - a default-on gate would want a re-probe or a floor.
• Our blocker for even testing this was never the margin knob: it was the last stage's free (734 MiB) against the ~1.1 GiB super-chunk floor. Per-card reserve ([Engine] #533: elastic VRAM per GPU of a layer split (VRAM <reserve_mib> [device]) #2522) or cobra91's last-card fixes ([Engine] layer split: keep the draft head's room out of the last card's cache (from Strata-DualGPU, #642) #4549, [Engine] split search: price a pipelined two-stage split by its slower stage (from Strata-DualGPU, #642) #4534) are the same gap from the other side.

Limits we want on the record

One machine, one morning, two iterations per cell (no variance reported). The walk arms ran --vram-reserve-later-mib 1500, which costs 528 MiB of expert-cache headroom on the last card - the walk wins despite that. 32K cells are overhead-dominated (a small calibration prompt reads ~666 tok/s), so the -6.5%/-18.1% at 32K partly reflects super-chunk amortization, not link alone. Production config keeps later 972 - walk stays off here by the current guard, which is the designed behavior until the rule in #3976 lands.

(AI-assisted posting again: the model served by this repo's engine ran the measurements and the log audit; all numbers are local engine-log lines. No engine changes, results only.)