Repository navigation
fix: overlay shadowing of huggingface-hub/transformers, TensorFold routing, FP8 on Mac, Flint message - #108
Merged
Conversation
…l type flint not supported' Recursive's MLX INT4 export stores int8 embeddings in a layout only their own engine reads, and older MLX stacks lack the architecture. The worker now says so and points at the Flint GGUF.
…hub / transformers; stock-loadable models stay off TensorFold; FP8 is flagged on a Mac The overlay guard only covered mlx / mlx-lm / mlx-metal, so an overlay holding huggingface-hub 1.12 under a 2.2 dist-info made transformers refuse to import and turned MLX off for every model. huggingface-hub, transformers, tokenizers, safetensors and hf-xet now move aside unless the overlay holds exactly the bundled version. A TensorFold-exclusive family is exclusive only when the app's mlx-lm lacks the architecture (mlx-lm 0.32 loads Ternary Bonsai 2). A compressed-tensors FP8 checkpoint is marked unusable on macOS.
…acts on them llama.cpp uses both (retry with fewer layers; --flash-attn). MLX and vLLM record Fit in memory without using it, and Fused attention reaches only a non-Native MLX cache strategy, so the switches did nothing on most models.
…ot the whole folder A repo folder holding BF16, F16 and Q8_0 of a 24B model is 148 GB on disk, so the launch screen showed 148.9/64 GB of RAM and the load check warned. The library entry now carries loadSizeGb (the pinned or largest model file, all shards) and the launch estimates use it.
…esident memory The warm-pool budget and the loaded-model figure the image and video gates use summed every quant in a GGUF repo folder (148 GB for one 24B model), so a loaded chat model could make those gates refuse.
…runs The first workspace response arrives with an empty library and libraryStatus 'scanning' while the scan is still going, and nothing refetched, so a window opened right after launch showed 'No models found' in My Models and the chat picker until the user reloaded it.
… load at a time; TensorFold toggle names DFlash 2 - model_has_mtp_tensors could not see through a Hugging Face cache folder (models--owner--name), returned None, and routing fell back to a catalog name match: a library item matched to Qwen/Qwen3.5-4B sent Qwen3-4B-bf16 to MTPLX, which refused it (no MTP head). It now resolves the newest snapshot and reads an unsharded model.safetensors header. - Two model loads could overlap: the second replaced the engine and orphaned the first's process (a 21 GB TensorFold worker kept running and slowed the next model). Loads and unloads now take a lock. - The TensorFold toggle says it carries DFlash 2 / MTP drafting; those models show no separate DFlash switch.
…rom the Dashboard A parked 21 GB TensorFold worker was invisible as stale and could not be stopped from the app. The process table gets a Stop button on model processes this app started that serve nothing, and a Stop idle processes button that unloads parked models and stops every unused model process now (no grace window). The backend refuses the loaded model's process (unload the model instead) and anything that is not a model process under this backend. Endpoints: POST /api/models/processes/stop and /api/models/processes/stop-idle.
…kes a grid column
…it instead of queueing The load lock added earlier made every repeat request wait its turn, so a model that was slow to start (a 27B TensorFold cold start) turned each further click or chat message into another full reload in sequence, and Unload itself waited behind them. Now: unload ends the load in flight (and queued loads give up), a request for a different model aborts the one running, a repeat of the running request waits for it and reuses the result, and an aborted load does not fall back to another engine.
…or streams; per-request enable_thinking for llama.cpp
…epo in Disk usage
…through the cache API
…llow and disk-usage repo delete
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found testing the v0.9.8 build on a Mac that had earlier Setup installs.