Eager NATIVE_OPTIMIZED: keep Q8_0 matmul weights packed (pre-transpose marker) so gemma fits + runs fast on the SL2610

[michalharakal]

Summary

The eager (OptimizedLLMRuntime DIRECT) Kotlin/Native path can't keep Q8_0 matmul weights packed, because linearProject transposes the weight and ops.transpose handles the K-series (Q4_K/Q5_K/Q6_K) but not Q8_0. This forces FunctionGemma's tied Q8_0 token_embd/output to dequant to FP32 (2×~0.67 GB), which OOMs the 1.9 GB Astra Machina SL2610 board. Result: the pure-SKaiNET eager LLM can't run on-device; only the IREE f16 vmfb path does.

This is the "pre-transposed marker" follow-up already noted in llm-core/.../transformer/LinearProjection.kt (Solution C / ISSUE-skainet-8b-oom.md).

Repro

GemmaQ5KPackedParityTest (host, -PincludeIntegration) with output/lm_head packed as Q8_0BlockTensorData (drop it from the isEmbed FP32 branch in GemmaPackedWeights.convertGemmaWeightsPacked + add Q8_0 to packGemmaKQuant):

java.lang.ClassCastException: class java.lang.Byte cannot be cast to class java.lang.Float
  at sk.ainet.lang.tensor.data.DenseTensorDataFactory.init(DenseTensorDataFactory.kt:539)
  at sk.ainet.exec.tensor.ops.DefaultCpuOpsBase.transpose(DefaultCpuOps.kt:632)
  at sk.ainet.lang.nn.transformer.LinearProjectionKt.linearProject(LinearProjection.kt:34)
  at sk.ainet.lang.nn.transformer.MultiHeadAttention.attentionImpl(...)   # also Q8_0 attn weights

linearProject = ops.matmul(input, ops.transpose(weight)) — transposing a Byte-backed packed tensor falls into the generic FP32 DenseTensorDataFactory path and crashes. K-series survive because they have a handled transpose; Q8_0 does not.

Root cause

• ops.transpose (engine, DefaultCpuOps) has no path for Q8_0TensorData (Byte-backed) — and broadly, linearProject should not be transposing a packed weight at all when the packed layout is already [out, in] block-major (the kernel's expected order).
• GemmaPackedWeights.convertGemmaWeightsPacked therefore FP32s token_embd/output; with tying, that's two ~0.67 GB FP32 matrices in opposite orientations.

Proposed fix (the actual Tier 2)

1. Pre-transposed marker on packed weights: mark a NATIVE_OPTIMIZED-packed weight as already [out, in] block-major; linearProject reads the marker and skips ops.transpose, dispatching straight through DefaultCpuOps.chooseQuantizedMatmulHeap (which already routes Q8_0TensorData → Q8_0 kernel). (Alternative/also: add Q8_0 support to ops.transpose — engine dependency.)
2. Pack Q8_0 matmul weights in packGemmaKQuant (generalize the relayout to 32-elem/34-byte blocks → Q8_0BlockTensorData); drop output from the isEmbed FP32 branch.
3. Row-dequant the MAIN token_embd gather so the embedding also stays Q8_0 (today only the per-layer PLE embed implements RowDequantSource; ops.gather on a packed tensor otherwise materializes full FP32). Then the tied embed/lm_head share one ~178 MB Q8_0 tensor.

Footprint ~1.34 GB → ~178 MB, and the lm_head runs on the NEON Q8_0 kernel.

Target / proof it's right

The upstream Synaptics sl2610-examples/Function_calling runs the identical GGUF via llama.cpp at ~9.7 tok/s on the 2-core Cortex-A55 — keeping the embedding Q8_0 packed (row-dequant gather + quantized lm_head). That's the goal for the eager path.

Notes for whoever picks this up

• Validate on the host GemmaQ5KPackedParityTest first (compiles from source; prints flush; no 1.9 GB OOM) before any board cycle.
• Kotlin/Native caches klibs by coordinate — same-version publishToMavenLocal republishes link STALE klibs. Clear ~/.konan/*/klib/cache + ~/.m2/.../<ver> + ~/.gradle/caches/modules-2/.../<ver> between republishes, or use a unique version. Verify with grep -a <marker> <binary>.kexe (ELF, uncompressed — valid; the .klib is a zip, grep is invalid).
• Cortex-A55: NEON+fp16+dotprod, no i8mm (don't add +i8mm → SIGILL).
• Board harness ready in SKaiNET-embedded/sl2610-function-calling: voicecc gen-native (eager load(NATIVE_OPTIMIZED) + greedy decode, parity vs llama-ref), linuxArm64 deps + linkerOpts(native/prebuilt/libskainet_kernels.aarch64.a) (board-built NEON archive). It builds/links/loads today; only OOMs at weight load on the unpatched converter.

🤖 Generated with Claude Code

[michalharakal]

Update: lm_head half implemented + host-verified ✅

The Q8_0-packed lm_head path is done and proven correct on the host.

Engine (SKaiNET branch fix/q8_0-lazy-transpose, 439bf919): added the missing is Q8_0TensorData -> case to DefaultCpuOps.transpose (rewraps packedData with flipped shape, like the K-series) — this was the source of the Byte cannot be cast to Float crash.

Transformers (branch fix/gemma-board-embed-nocopy, f94ce6c):

• packGemmaKQuant packs Q8_0 (32-elem/34 B → Q8_0BlockTensorData; relayout generalized with a blockSize param).
• output/lm_head dropped from the isEmbed FP32 branch → stays packed (runs on the NEON Q8_0 kernel).
• token_embd FP32 wrapped no-copy via DenseFloatArrayTensorData (avoids the ctx.fromFloatArray second-buffer double).

Verified: GemmaQ5KPackedParityTest via composite -PuseLocalSkainet=true (both repos compiled from source) — eager load(NATIVE_OPTIMIZED) decodes byte-identically to the FP32 baseline, lm_head packed Q8_0, no ClassCastException. Tied embed/lm_head footprint drops ~1.34 GB → ~0.76 GB (embed FP32 ~0.67 GB + packed Q8_0 lm_head ~0.09 GB).

Remaining for full board fit (the other half): row-dequant the main token_embd gather (today only the per-layer PLE embed implements RowDequantSource) so the embedding also stays Q8_0 — drops the last ~0.67 GB.

Board-demo note: consuming these local fixes on the SL2610 is blocked by tooling, not code — Gradle doesn't honor mavenLocal() over Central for the BOM-managed KMP module (the local klib has the change; the linked binary doesn't), and a composite-for-app hits Kotlin-version skew across the three repos. Cleanest path is to land these branches → CI publish → consume normally.

🤖 Generated with Claude Code

[michalharakal]

Engine transpose fix completed for all packed matmul dtypes: SKaiNET#736 (Q8_0, merged) + SKaiNET#737 (Q4_0 + a regression test transposing all 7 packed types: Q4_K/Q5_K/Q6_K/Q5_0/Q5_1/Q8_0/Q4_0). Bf16/Ternary are non-crashing in the generic path and not matmul-dispatched, so out of scope.

[michalharakal]

On-board: weight fix confirmed working; full decode needs the KV-cache cap (#180) + published libs

A composite build of the full fixed stack (engine #736+#737 + transformers #179) was run on the SL2610:

• Load now fits: FunctionGemma loads to a stable ~1.06 GB resident — vs the prior ~1.5 GB OOM-at-load. So the packed-Q8_0-lm_head + no-copy-embed fix drops the footprint on real hardware. ✅
• The decode then OOM'd only when the first forward allocated the KV cache sized for maxInferenceLen=4096 (~0.4 GB). Addressed by feat(gemma): optional maxInferenceLen on load() to cap KV cache on constrained devices (#178) #180 (load(maxInferenceLen = 32)) — for a ~13-token tool-call prompt this shrinks the KV cache ~100×.

Remaining: (1) merge #737 + #180; (2) the token_embd main-embedding row-dequant (drops the last ~0.67 GB FP32 embed); (3) publish the fixes so the SL2610 app consumes them normally — local consumption (mavenLocal / unique-version full publish / composite) all hit Gradle-KMP tooling walls (metadata-compile skew, coordinate-cache, substitution mismatch), so CI-published versions are the clean path.

Net: fix is host-correct (byte-identical decode) and footprint-proven on-board; only packaging remains.

[michalharakal]

All code merged ✅ — remaining is release coordination

• Engine develop: #736 + #737 merged — ops.transpose now lazily handles all packed matmul dtypes (Q4_K/Q5_K/Q6_K/Q5_0/Q5_1/Q8_0/Q4_0) + regression test.
• transformers develop: fix(gemma): keep tied Q8_0 lm_head packed in eager NATIVE_OPTIMIZED path (#178) #179 (packed Q8_0 lm_head) + feat(gemma): optional maxInferenceLen on load() to cap KV cache on constrained devices (#178) #180 (load(maxInferenceLen) KV-cache cap) merged.

The fix is host-correct (byte-identical eager decode) and on-board footprint-proven (~1.5 GB OOM-at-load → stable ~1.06 GB).

⚠️ Release sequencing: transformers develop still pins skainet = 0.30.0, which lacks the ops.transpose Q8_0/Q4_0 cases. The next transformers release MUST consume an engine build that includes #736+#737 (publish engine first → bump the skainet pin) or the eager Q8_0 lm_head path will Byte→Float ClassCastException at runtime.

Still open in this issue: the token_embd main-embedding row-dequant (the other ~0.67 GB FP32 embed) for full board headroom; and the end-to-end on-board decode validation once the libs publish (then the SL2610 app passes maxInferenceLen = 32).