[dansupergameprogrammer]

Steps D0 and D1 of the decode-threading plan (rev 1.2, approved). This is meant to ship as 1.11.0. The version number is not bumped here, and nothing is released.

Before / After

Before: decode mat-vecs run on one thread. A host's sslm_parallel_for hook is used only by the final logits step. Every per-layer projection in decode, and in a prefill that admits one token, runs on the calling thread.

After: a host can opt in by setting bit 1 of sslm_parallel_for.reserved, the new SSLM_PARALLEL_FOR_MATVEC. With the bit set, each one-row (M = 1) projection splits its output rows across the hook's run. That covers each decode layer's five groups (q alone, k + v together, o alone, gate + up together, down alone), and each of the seven projections of a prefill that admits exactly one token. Every row is still one exact integer sum computed on one thread, so the split is exact.

Default: off. Without the bit nothing changes, and a 1.10.0 host behaves exactly as before. The plugin's own default is not set here; the owner sets it after the box measurements (see Still owed).

How

• Shared split code. A new internal source, src/forward/parallel_split.{h,cpp}, holds the column split, the run-exactly-once runner and a snapshot of the per-sequence saturation counters. The logits step now uses the same split. Its partition is unchanged.
• When a projection splits. The task count is the smallest of max_tasks, ceil(rows / 64) and floor(weight bytes / 256 KiB). Rows per task are then rounded up to 64, so 256 KiB caps the task count; it is not a floor on each task, and the last task can be smaller (896 × 896 at 4 tasks: 320/320/256 rows, the last 229,376 bytes). A group that gets fewer than two tasks stays serial. Qwen2.5-0.5B with 4 tasks makes 97 run calls per decode token (k + v stays serial).
• Setters. Both setters accept bit 1 and still refuse every other bit. The GPU setter accepts the bit and ignores it.
• Retry after a hook failure in decode. If run breaks its exactly-once contract inside a decode layer, the call fails with SSLM_INVALID_ARGUMENT. The sequence rests at the start of that layer with that layer's saturation counts put back, and the next call resumes there.
• One behaviour change for every host, bit or not. A prefill that fails before admitting a token now leaves the saturation counts as they were before the call.
• C++ exports. New threaded overloads of RunLayerLoop and RunLayerLoopChunkBatched. The existing symbols stay. No struct, status, C function or save format changes.
• Tests and CI. The tests read fixtures generated in CI by tools/gen_decode_threading_fixture.py; they are never committed. Every CI leg that runs a suite binary gains a step that generates them. A missing fixture fails the tests rather than skipping them.
• Bench. tools/decode_threading_bench.cpp is the tool for the box measurements.
• Where to read more. Details, and the 19 places this build differs from the plan, are in docs/decode-threading-d0d1-progress.md.

Code review (HOLD at 5043ce4) and fixes

The review passed the core threading on Windows MSVC: output identity, retry state, allocation, ABI and seams. On Windows, 133,887 checks with 0 failures, forced SSE2 and AVX2 green, and the WIN32 setter cell passed. It raised four findings, each fixed in its own commit:

• P1 (ship-blocking), 9677431. tools/build_adapter_dump.bat named parallel_split.cpp without %ENG%. Grepping for the same defect found 26 more MSVC build lists (red-suite scripts under tests/, the t2922 PowerShell runners, the two t2132 diagnostics) that omitted the file entirely and so failed to link. All are fixed. No CI step or checker compiles these scripts.
• P2 (PD-1), 6671e43. Cell 2.1 encodes the owner's ruling. The hook reads the live counters at the failure. At q and k + v nothing has moved yet, so those cases prove no drift and retry identity and claim no restore. At o, gate + up and down, the checked counters are proved to have risen before the failure, then restored. The 11 mutants that touch 2.1 were re-run: 11 killed.
• P3, 5025769. docs/api.md, parallel_for.h and the CHANGELOG now say what the split does (above).
• P4, c5efc72. F-QK is host-dependent. numpy dispatches float64 np.exp on the CPU (its own AVX-512 exp here, the C library's elsewhere), and F-QK's QKC1 table is exact in the float calibration peaks. Only that section moves; F-DEF and fnoclamp have no QKC1 table and match everywhere. Nothing in 1.10.0 changed the generator, and no engine code runs in it. This host at this head still writes 27e82bcb…. 6.3 and 9.1 against v1.9.0 were re-run on the fresh fixtures and on both local F-QK variants (27e82bcb…, 623d3efe…): byte-equal.

Bit-identity evidence

Checked on Linux with GCC 13 and Clang 18, Release, on an AVX-512 host, at the review-fix head. Full record: docs/decode-threading/review-fixes.txt; the move onto 1.10.0 is in docs/decode-threading/rebase-1.10.0.txt.

• Suites. The auto suite and the SSE2-, AVX2- and AVX-512-forced suites: 0 failures on both compilers. superslm_tests has 123,814 checks (main 121,655), with 2,152 D1 checks after the PD-1 change. The scalar-tier decode-threading binary (2,159 checks) and the per-token allocation binary (84 checks) also pass with 0 failures.
• Output digests. All ten digest legs (five tiers × two compilers) give GLOBAL f740f833…4ae5acc, which is main's hash. The whole output matches main's byte for byte on every leg.
• With threading on. Tokens and save blobs with the bit on equal the serial run at max_tasks 1, 2, 4 and 8. The tests also check each row against the scalar reference.
• Other checks. The fp-free scan passes on both compilers (18 objects, 0 rejected). pytest tests/ci gives 915 passed, 1 skipped, the same as main. The CI-claims, present-tense, GPU-guard parity, linkage, isolation, forward-leaf and site-census checks all pass.
• Mutation tests. 23 of 23 planted bugs are caught. They ran on the same source before the rebase, and the rebase did not change the source. The 11 that touch cell 2.1 were re-run against the PD-1 cell: 11 of 11 caught.
• Sanitizers. ASan+UBSan and TSan reported 0 errors on the same source before the rebase.

Indicative cloud speed

Decode tokens/s with the bit on are 1.27–1.57x the rate with it off. This was measured on a shared 4-vCPU cloud Xeon with a synthetic 2-layer artifact at 0.5B width. Indicative only. Not measured on the owner's desktop, and not a plan reading.

Still owed

• On Windows (the owner's machine): run tools\build_adapter_dump.bat and at least one of the other patched MSVC lists; the WIN32 setter gate and the Windows suites at the new head; optionally the v1.9 probe on the Windows-generated F-QK (31dd0c5b…).
• A confirming review of the four fixes.
• D2 box timing (B0–B2) on the owner's desktop with the real model. The 256 KiB minimum-work constant is also settled there.
• After that, the owner sets the plugin default from B3's off/on numbers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XfuJLAZ6EFSt7mB85oNm68

---

Generated by Claude Code