Repository navigation
vulkan : add TileK-first activation layout for Adreno coopmat matmul (Q4_0) - #10
Closed
Raman Shinde (Raman-Raje) wants to merge 79 commits into
Closed
Raman Shinde (Raman-Raje) wants to merge 79 commits into
Raman Shinde (Raman-Raje) wants to merge 79 commits into
Conversation
Sync with mainline
Sync with main
Sync with main
Sync with main
Sync with main
Sync with main
Sync to main
SearchInput already renders its own clear button. Hide the browser's native search cancel button to avoid showing both controls. Fixes ggml-org#28627 Signed-off-by: aiqubits <aiqubits@hotmail.com>
…gml-org#30854) * hex-build: bump toolchain, enable v85 * hex-build: bump toolchain to v0.8 in the CI and build scripts * hex-build: add a little python script for swiv needed on some iot devices * hex-build: fix flake8 warnings * hex-arch: update version cap to v85 * hex-inspect: flexible TOOLS_ROOT * hex-build: update to snake_case naming * hex-build: update leftover old toolchain refs * hex-target: update build and run scripts to support ubuntu with adb transport
…gml-org#30818) * tests : make MoE router selection independent of the backend noise The router scores are in the range [0, 1), so a random draw can leave the last expert (or expert group) still selected and the first one dropped tied within the noise between the backends. They then route a token to different experts and the logits differ by 4e-3 instead of the usual 5e-7. Beam is the only arch here with a real top-k (8 experts, 2 used) and a group top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie. Fill the selection bias with a ramp of step 2 to give every expert a score range of its own, which turns the top-k into a function of the expert index alone. The bias affects only the selection, the weights still come from the unbiased probs, so the MoE output keeps its magnitude. Beam now matches the CPU reference within 4e-7 for all of 160 seeds. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : tell the CPU and the GPU of an Apple SoC apart in the device table Both devices report the same description ("Apple M2 Ultra" here), so the device column of the test table had two indistinguishable rows and a failing row did not say whether the Metal device or the host CPU produced it. Add the device name, which is also what -b takes, unless the description already is the name. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
abi-compliance-checker was running in strict mode unconditionally, rather than honoring the CLI flag.
…gml-org#28554) * server : pass id_slot through the Anthropic /v1/messages conversion server_chat_convert_anthropic_to_oai copies a small whitelist of common params from the Anthropic request into the internal OAI-style body and left id_slot out. Every other completion endpoint honours id_slot; a proxy that pins a session to the slot holding its KV cache needs the same on /v1/messages, or it silently falls back to automatic slot selection and, on hybrid/recurrent models, pays a full re-prefill. Adds a test asserting a request with id_slot set is served by that slot, observed via GET /slots. * server : shorten comments * rm test --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* spec : save the dynamic ngram cache file * save the cache at the common_speculative_impl_ngram_cache destruction * remove save_static/save_dynamic flags as well as the relevant TODO * if the dynamic cache file does not exist, start with an empty cache that is written on exit. if it exists but cannot be read, abort instead of overwriting it. * spec : fix draft_one not updating the ngram cache history correctly * common_ngram_cache_update received only the new tokens with none of the tokens before them causing ngrams ending at the new token to not being formed, leading the context cache to only ever hold the prompt * fix: pass the full history to common_ngram_cache_update with nnew set to the number of new tokens the way llama-lookup does and also reset cache_size to 0 on each new request --------- Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
* ggml-zdnn: add scratch buffer pool Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * ggml-zdnn: implement fattn, ditch scratch slots in favour of scratch buffer pool Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: add note about fattn Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
The __ARM_NEON branch defines both GGML_CPU_COMPUTE_* macros but selects only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable software conversion while neon_compute_fp32_to_fp16, a single fcvt, stays defined and referenced by nothing. __POWER9_VECTOR__ and __riscv_zfhmin select both. Add an #error pair that refuses a branch defining a COMPUTE macro without selecting it. That is how this stayed unnoticed: the fallback silently supplies a correct software implementation, so the only symptom is a hardware instruction nothing ever reaches. The FP16_TO_FP32 half carries a !defined(__F16C__) carve-out, because PR 30343 keeps that direction on the lookup table on purpose. Replace the comment above the fallback: it singled out NEON and POWER9 and pointed at a selection that now happens in the branch itself.
…gml-org#30850) * opencl: fix OOB read in `mul_mv_mxfp4_f32_flat` * opencl: remove large-N guard that forces CPU weight placement * this condition claims that large batch produces incorrect results on adreno and cannot be reproduced * model loader uses N512 to probe op support, resulting in cpu fallback * thus this condition does not seem necessary * opencl: cap num of subgroups by workgroup size for q4_0 mc3 Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> * opencl: add small batch variant for q4_k dp4a GEMM Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> * opencl: add a narrow variant of mul_mm_f16_f32_l4_lm Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> * opencl: add a one subgroup variant for gemv_noshuffle_q1_0_f32 for big M Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> --------- Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
ggml-org#30865) * model : drop the redundant precision overrides in the Kolibri-1 router Follow-up to the review in ggml-org#30673: the router matmul works with the default precision, so both ggml_prec_set_* calls can go away. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * cont : fix comment Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* metal : optimize argmax and pool_1d Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * tests : add pool_1d perf cases Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * cont : minor fix
* mtmd : read LFM2 tiling params from GGUF metadata Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp with the previous constants as defaults. do_image_splitting=false maps to min_tiles=max_tiles=1 (no tiling). Rebased version of ggml-org#25524, original implementation by Michael DM (@michael-dm). Co-authored-by: Michael DM <usable_siren180@simplelogin.com> * fix test --------- Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
…atibility (ggml-org#30863) * server: include output_index in Responses API SSE events * server: reuse Anthropic streaming code for fix output_index * server: remove unnecessary generated_tool_call_ids and oai_resp_tool_calls_started
… child, the response wait could cause a delayed cancellation (ggml-org#29707) * Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation * tests: satisfy router disconnect type check * server : remove timing-sensitive router cancellation test Assisted-by: Codex
Branch: GraniteSpeculatorSupport AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
) * sycl: fix CONV_2D wrong results for non-contiguous input The kernel indexes the `input` as a contiguous [W, H, C, N] tensor, but supports_op claimed the op for any layout. A non-contiguous (e.g. `CWHN` permuted) input was read with the wrong element offsets, silently producing wrong results. Gate the CONV_2D op on `input` and `kernel` contiguity, matching the CUDA backend contract, and assert input contiguity in the op. `test-backend-ops -b SYCL0 -o CONV_2D`: goes from 16 failures to 0, none. ```text 2006/2006 tests passed Backend SYCL0: OK Backend 2/2: CPU Skipping 2/2 backends passed OK ``` Assisted-by: pi (unsloth/Qwen3.8-27B-GGUF:Q4_K_M via llama.cpp) * sycl : CONV_2D cwhn=1 support Make the SYCL backend compute `GGML_OP_CONV_2D` correctly for `cwhn=1` (CWHN channel-order) inputs and kernels, so the previously failing test- backend-ops CONV_2D cases pass on SYCL instead of being skipped or miscomputed. - Implement CWHN layout support in `ggml/src/ggml-sycl/conv2d.cpp` (`ggml_sycl_op_conv2d`). - Update SYCL `supports_op` for `GGML_OP_CONV_2D` in `ggml/src/ggml-sycl/ggml-sycl.cpp` to accept both layouts. Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF via llama.cpp) * sycl docs : update ops for SYCL
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This adds a TileK-first activation layout for Adreno GPUs. In this layout, K is stored in contiguous 16-wide strips, so the coopmat matmul can load B directly without staging it in shared memory.
Requirements