Repository navigation
models: Flint (FlintForCausalLM) on ra/main - #6
Merged
Merged
Conversation
* llama : scale moe down projection per expert token Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo the scale on the output with the same exact power-of-two factor. Handle POOL_1D in the Meta split-state logic so a split along axis 0 becomes a replicated per-device partial result, enabling tensor parallelism. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * cont : simplify comments Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * llama : refactor amax src1 scaling result Return the rescaled activation and the optional scale tensor through a small struct instead of an output pointer. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * cont : add TODO + clean-up (cherry picked from commit 8baabd5)
(cherry picked from commit 1b44e30)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Flint (
FlintForCausalLM,unsloth/flint-GGUF) to the fork. Three commits for Flint, plus one CI fix:2878f634cherry-picks upstream llama : scale moe down projection per expert token ggml-org/llama.cpp#30861. Flint's MoE path callsbuild_amax_src1_f16, which isn't in the fork yet. It also replaces the oldggml_prec_set_src(F32)line for Mistral4 with upstream's graph-level rescale, so Mistral4 and Mellum change behaviour here.808825f9cherry-picks the Flint PR, models: Add Flint model (FlintForCausalLM) ggml-org/llama.cpp#30928 at1b44e307. That PR is approved upstream but not merged. Conflicts were resolved by adding onlyLLM_ARCH_FLINT, not the upstream-only archs that sat next to it.cb858de7is a port fix. Upstream adds logits to the graph in a sharedbuild_logits(), and the fork's model files still do it themselves. Without this line every Flint decode asserts atllama-context.cpp:1956. Drop this commit when the fork syncs pastbuild_logits().Both cherry-picks carry
-xtrailers, so the next upstream sync can drop them.Tested on an M4 Pro (48 GB), Metal:
test-llama-archs -a '^flint$'test-llama-archs, excludingdeberta*RecursiveSI/flintllama-serverchat, UD-Q4_K_XL-c 1024, 1,600 tokenstool_callsTwo known gaps. Flint's embedded chat template uses
replace("\x00", ""), and the Jinja lexer here and upstream rejects\xescapes, so--jinjafails at start-up unless you pass a template with that filter removed via--chat-template-file. That fix belongs upstream. Separately,deberta-v2aborts in thetest-llama-archsmodel generator with a missingrelative_pos_maxkey. Plainra/main(a8ea71dc) fails the same way, so that's an existing problem.d37c473efixes thepre-tokenizer-hashescheck, which has been red on the fork sincec6aa47c6. The GLiNER2.5 hash was hand-added toconversion/base.py, so the generator dropped it on every run. It's now registered inconvert_hf_to_gguf_update.pywith its source repo,fastino/GLiNER2.5-Decide, which I confirmed by recomputing the hash. Running the generator twice gives the same file. The flake8Lintfailures are also older; they're inconversion/bert.pyandgguf-py/gguf/constants.pylines fromc6aa47c6.