Skip to content

models: Flint (FlintForCausalLM) on ra/main - #6

Merged
Siddhesh2377 merged 4 commits into
ra/mainfrom
siddhesh/flint-support
Oct 21, 2026
Merged

Siddhesh2377 merged 4 commits into
ra/mainfrom
siddhesh/flint-support

Conversation

@Siddhesh2377

@Siddhesh2377 Siddhesh2377 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Adds Flint (FlintForCausalLM, unsloth/flint-GGUF) to the fork. Three commits for Flint, plus one CI fix:

  • 2878f634 cherry-picks upstream llama : scale moe down projection per expert token ggml-org/llama.cpp#30861. Flint's MoE path calls build_amax_src1_f16, which isn't in the fork yet. It also replaces the old ggml_prec_set_src(F32) line for Mistral4 with upstream's graph-level rescale, so Mistral4 and Mellum change behaviour here.
  • 808825f9 cherry-picks the Flint PR, models: Add Flint model (FlintForCausalLM) ggml-org/llama.cpp#30928 at 1b44e307. That PR is approved upstream but not merged. Conflicts were resolved by adding only LLM_ARCH_FLINT, not the upstream-only archs that sat next to it.
  • cb858de7 is a port fix. Upstream adds logits to the graph in a shared build_logits(), and the fork's model files still do it themselves. Without this line every Flint decode asserts at llama-context.cpp:1956. Drop this commit when the fork syncs past build_logits().

Both cherry-picks carry -x trailers, so the next upstream sync can drop them.

Tested on an M4 Pro (48 GB), Metal:

Check Result
test-llama-archs -a '^flint$' 4/4 OK
full test-llama-archs, excluding deberta* 514/514 OK
tokenizer vs HF RecursiveSI/flint 10/10 lines identical
llama-server chat, UD-Q4_K_XL correct output, reasoning split out, stops on EOG, about 54 tok/s
3,275-token prompt (past the 513 SWA window) recalled a code from the start exactly
context shift, -c 1024, 1,600 tokens no crash, output stays coherent
perplexity, wikitext-2, ctx 512, 20 chunks 22.81 ± 1.29
tool call parsed into tool_calls

Two known gaps. Flint's embedded chat template uses replace("\x00", ""), and the Jinja lexer here and upstream rejects \x escapes, so --jinja fails at start-up unless you pass a template with that filter removed via --chat-template-file. That fix belongs upstream. Separately, deberta-v2 aborts in the test-llama-archs model generator with a missing relative_pos_max key. Plain ra/main (a8ea71dc) fails the same way, so that's an existing problem.

d37c473e fixes the pre-tokenizer-hashes check, which has been red on the fork since c6aa47c6. The GLiNER2.5 hash was hand-added to conversion/base.py, so the generator dropped it on every run. It's now registered in convert_hf_to_gguf_update.py with its source repo, fastino/GLiNER2.5-Decide, which I confirmed by recomputing the hash. Running the generator twice gives the same file. The flake8 Lint failures are also older; they're in conversion/bert.py and gguf-py/gguf/constants.py lines from c6aa47c6.

ggerganov and others added 3 commits October 22, 2026 00:58
* llama : scale moe down projection per expert token

Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo the scale on the output with the same exact power-of-two factor. Handle POOL_1D in the Meta split-state logic so a split along axis 0 becomes a replicated per-device partial result, enabling tensor parallelism.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : simplify comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* llama : refactor amax src1 scaling result

Return the rescaled activation and the optional scale tensor through a small struct instead of an output pointer.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : add TODO + clean-up

(cherry picked from commit 8baabd5)
@Siddhesh2377
Siddhesh2377 merged commit f1f50c0 into ra/main Oct 21, 2026
11 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants