fix: overlay shadowing of huggingface-hub/transformers, TensorFold routing, FP8 on Mac, Flint message

.gitignore

@@ -1,10 +1,12 @@
 __pycache__/
 *.pyc
+.venv
 .venv/
 *.egg-info/
 build/
 dist/
 .DS_Store
+node_modules
 node_modules/
 src-tauri/target/
 src-tauri/resources/embedded/

CHANGELOG.md

@@ -19,6 +19,8 @@ First release since v0.9.4; versions 0.9.5 to 0.9.7 were internal. Full notes in
 - New models: LFM2.5 24B-A2B (MLX 4/8-bit and GGUF), Muse Glimmer 30B GGUF (vision, DFlash 2 sidecar), Kolibri 1 GGUF Q3_K_M, Qwen3.8 Flash Next GGUF, Qwen3.6 35B-A3B MTP through TensorFold, Qwen-Image-2.1 Turbo and FLUX 3 Image Klein through mflux.
 - Image input for MLX vision models loaded through mlx-vlm (Gemma 4 on Apple Silicon).
 - Memory and speed estimates rebuilt for chat, image and video: real KV-cache layout per architecture, memory pressure from available memory, decode speed from this machine's history or its memory bandwidth, an advisory check before a chat model loads, and engine-aware image and video estimates (FU-110).
+- Dashboard process list: Stop per leftover model process and a Stop idle button; unloading cancels a load in flight and a newer load replaces the one running.
+- Disk usage breaks the shared Hugging Face blob store down by repo, and a cached repo can be deleted there through the cache API.
 - `release-constraints.txt` pins every package in release builds to the tested versions.
 
 ### Fixed

RELEASE_NOTES_v0.9.8.md

@@ -7,6 +7,7 @@ The first release since v0.9.4. Versions 0.9.5 to 0.9.7 were internal builds, so
 - **DFlash on Apple Silicon finally runs.** Since dflash-mlx 0.1.5, every DFlash reply had quietly fallen back to normal decoding while the load note still said DFlash was active. Fixed: Qwen3-4B now writes at 113 to 138 tokens per second against 53 without it, with identical output.
 - **Newer MLX underneath.** Releases now ship mlx 0.32, mlx-lm 0.32 and transformers 5.20. Setup installs resolve mlx-vlm 0.7.10 and diffusers 0.41. Current Gemma 4 conversions load (the old mlx-vlm could not read them), and every release build pins the exact versions it was tested with.
 - **Images in chat on Apple Silicon.** Vision models loaded through mlx-vlm (Gemma 4) now see attached images. Before, the image was always dropped.
+- **Models stop loading when you ask.** Unload cancels a load in flight, and picking a different model replaces the one that is loading instead of queueing behind it. The Dashboard lists leftover model processes with a Stop button each and a Stop idle button, so a stuck worker can no longer hold 20 GB of memory with no way out. Send is disabled while a model loads, so a message can no longer go to the previous model.
 - **Honest memory and speed estimates.** The launch screen works out the real KV-cache size for each model's architecture and estimates speed from your own past runs or your machine's memory bandwidth. It warns before you load a chat model that may not fit. Image and video estimates know which engine they run on and count a chat model that is already loaded.
 
 ### New models
@@ -31,7 +32,7 @@ The first release since v0.9.4. Versions 0.9.5 to 0.9.7 were internal builds, so
 ### Faster generation
 
 - **DFlash** (Apple Silicon) fixed, as above.
-- **Thinking Off now works** for Qwen 3 / 3.5 / 3.6 / 3.8 on Apple Silicon. Before, they reasoned anyway and spent the reply budget on it.
+- **Thinking Off now works** for Qwen 3 / 3.5 / 3.6 / 3.8 on Apple Silicon, and the switch is sent to llama.cpp models too. Before, they reasoned anyway and spent the reply budget on it.
 - **TensorFold 0.6.6** (Apple Silicon): exact speculative decoding with no change in output. Install it from the launch settings.
 - **llama.cpp b11810** bundled: DFlash 2, EAGLE-3 and draft-free `ngram-mod`, picked automatically from the drafter files next to a GGUF.
 - **MTPLX 2.12**: reasoning now streams correctly.
@@ -54,7 +55,9 @@ A Voice group in the sidebar: speech-to-text and text-to-speech with Studio, Mod
 
 - **Chat > My Models lists chat models only.** 3D tool parts, text encoders, voice models and bare weight files no longer show up as chat models.
 - **Settings > Diagnostics is split into three tabs**: Diagnostics, Accelerators and Disk usage.
-- **Disk usage rows expand and can be deleted.** The shared Hugging Face `blobs` store is protected.
+- **Disk usage rows expand and can be deleted.** The shared Hugging Face `blobs` store is now broken down by repo, and any cached repo can be deleted from there; blobs another repo still uses are kept.
+- **Streaming follows the bottom.** In Chat, Compare and the HTML challenge the view now stays with the newest text until you scroll up. Before, one long chunk made it stop following, and Latest had to be pressed again and again.
+- **Multi-quant GGUF repos show the size of the one file that loads**, not the whole folder, in the library and the memory bar. A model that fits no longer looks too big to run.
 
 ### Fixes
 
@@ -68,6 +71,10 @@ A Voice group in the sidebar: speech-to-text and text-to-speech with Studio, Mod
 - **Dead catalog links** repaired; an audit script checks every row.
 - **Release builds** include the `dflash` package, the Setup installers and a pinned llama.cpp build, so GGUF models work out of the box.
 - **Tool use on Apple Silicon** no longer loops.
+- **Loading** a model no longer hangs forever when a second load is requested: repeated requests for the same model share one load.
+- **My Models and the chat picker** no longer come up empty while the first library scan is still running.
+- **MTPLX** is no longer chosen for models without an MTP head, and the TensorFold toggle names DFlash 2 and MTP drafting. A cold TensorFold start says it can take a few minutes.
+- **Extras overlay**: an older huggingface-hub or transformers left by a Setup install can no longer shadow the ones the app ships.
 - **MLX probe**: one stray kill of the startup check at launch no longer turns MLX off until you restart; it is retried once. The launch screen's context setting is capped at the model's own window, so a stored 262K no longer triggers a false memory warning on a 32K model.
 - **Hub listings** work on Python installs without a certificate bundle (Discover came back empty there).
 - **Downloads** skip weight shards that no index file uses, and the Gemma 4 entries are merged into one family.

backend_service/helpers/discovery.py

@@ -197,6 +197,14 @@ def _model_has_files(path: Path, pattern: str) -> bool:
 
 
 
+def _gguf_load_size_gb(directory: Path) -> float | None:
+    """Size of the one GGUF that loads from a multi-quant repo folder."""
+    from backend_service.helpers.text_gguf import gguf_load_bytes
+
+    size = gguf_load_bytes(directory)
+    return round(size / 1_000_000_000, 2) if size else None
+
+
 def _detect_storage_format(path: Path, *, name_hint: str = "") -> str:
     lowered_hint = f"{name_hint} {path}".lower()
     if path.is_file() and path.suffix.lower() == ".gguf":
@@ -483,6 +491,7 @@ def _discover_local_models(model_directories: list[dict[str, Any]], limit: int =
                         "backend": backend,
                         "modelType": model_type,
                         "sizeGb": _du_size_gb(child),
+                        "loadSizeGb": _gguf_load_size_gb(child) if file_format == "GGUF" and child.is_dir() else None,
                         "lastModified": time.strftime("%Y-%m-%d %H:%M", time.localtime(child.stat().st_mtime)),
                         "actions": ["Run Chat", "Run Server", "Cache Preview", "Delete"],
                         "directoryId": directory_id,

backend_service/helpers/persistence.py

@@ -11,7 +11,7 @@
 LEGACY_SEEDED_CHAT_IDS = {"ui-direction", "model-shortlist"}
 LEGACY_SEEDED_BENCHMARK_IDS = {"baseline", "native-34", "native-36", "native-44"}
 
-LIBRARY_CACHE_VERSION = 4  # 4: modelType "other" (non-chat models) added
+LIBRARY_CACHE_VERSION = 5  # 5: loadSizeGb (size of the one GGUF that loads)
 
 
 def _default_chat_variant() -> dict[str, Any]:

backend_service/helpers/quantization.py

@@ -14,6 +14,7 @@
 from __future__ import annotations
 
 import re
+import sys
 from typing import Any
 
 
@@ -51,7 +52,32 @@ def _quantization_algo_label(config: dict[str, Any] | None) -> str | None:
     return None
 
 
+def _is_fp8_compressed_tensors(config: dict[str, Any] | None) -> bool:
+    """A compressed-tensors checkpoint whose weights are FP8 ("float-quantized")."""
+    if not isinstance(config, dict):
+        return False
+    payload = config.get("quantization_config")
+    if not isinstance(payload, dict) or payload.get("quant_method") != "compressed-tensors":
+        return False
+    if payload.get("format") == "float-quantized":
+        return True
+    groups = payload.get("config_groups")
+    if isinstance(groups, dict):
+        for group in groups.values():
+            weights = group.get("weights") if isinstance(group, dict) else None
+            if isinstance(weights, dict) and weights.get("type") == "float":
+                return True
+    return False
+
+
 def _unsupported_mlx_quantization_reason(config: dict[str, Any] | None) -> str | None:
+    # On a Mac MLX is the only runtime, and it cannot dequantize FP8 safetensors
+    # (mlx-lm convert stops with "unsupported compressed-tensors format").
+    if sys.platform == "darwin" and _is_fp8_compressed_tensors(config):
+        return (
+            "This FP8 checkpoint (compressed-tensors) needs an NVIDIA GPU runtime such as vLLM; "
+            "MLX cannot read it. Use an MLX or GGUF build of this model on a Mac."
+        )
     algo = _quantization_algo_label(config)
     if not algo or algo not in _UNSUPPORTED_MLX_QUANT_ALGOS:
         return None

backend_service/helpers/text_gguf.py

@@ -192,3 +192,27 @@ def pick_model_gguf(directory: Path, model_files: list[Path]) -> Path:
         if first.exists():
             return first
     return chosen
+
+
+def gguf_load_bytes(directory: Path) -> int | None:
+    """Bytes of the GGUF llama.cpp would load from ``directory`` (all shards), or None.
+
+    A repo folder can hold many quants (a 24B model's BF16, F16, Q8_0 ... is 150 GB
+    on disk), but one file loads, so memory estimates must use that file.
+    """
+    try:
+        files = [f for f in directory.rglob("*.gguf") if not is_aux_gguf(f.name)]
+        if not files:
+            return None
+        files.sort(key=lambda f: f.stat().st_size, reverse=True)
+        chosen = pick_model_gguf(directory, files)
+        total = chosen.stat().st_size
+        match = _SHARD_RE.search(chosen.name)
+        if match:
+            stem = chosen.name[: match.start()]
+            for shard in chosen.parent.glob(f"{stem}-*-of-{match.group(2)}.gguf"):
+                if shard != chosen:
+                    total += shard.stat().st_size
+        return total
+    except OSError:
+        return None

backend_service/inference/_mtp.py

@@ -183,6 +183,37 @@ def _read_safetensors_index(model_dir: _Path) -> dict[str, str] | None:
     return weight_map if isinstance(weight_map, dict) else None
 
 
+def _snapshot_dir(directory: _Path) -> _Path:
+    """The newest snapshot of a Hugging Face cache repo folder, else ``directory``.
+
+    The library hands the engine ``models--owner--name`` (no weights in it), so
+    without this the probe finds no index and callers fall back to a name match,
+    which a mis-matched catalog name can get wrong.
+    """
+    snapshots = directory / "snapshots"
+    if not snapshots.is_dir():
+        return directory
+    try:
+        found = [d for d in snapshots.iterdir() if d.is_dir()]
+    except OSError:
+        return directory
+    return max(found, key=lambda d: d.stat().st_mtime) if found else directory
+
+
+def _read_single_safetensors_keys(model_dir: _Path) -> dict[str, str] | None:
+    """Tensor names of an unsharded ``model.safetensors`` (its JSON header), or None."""
+    weights = model_dir / "model.safetensors"
+    try:
+        with weights.open("rb") as fh:
+            header_len = int.from_bytes(fh.read(8), "little")
+            if not 0 < header_len < 100 * 1024 * 1024:
+                return None
+            header = _json.loads(fh.read(header_len))
+    except (OSError, ValueError):
+        return None
+    return {name: "model.safetensors" for name in header if name != "__metadata__"}
+
+
 def model_has_mtp_tensors(path: str | None) -> bool | None:
     """Authoritative MTP-head probe via local model files.
 
@@ -225,10 +256,12 @@ def model_has_mtp_tensors(path: str | None) -> bool | None:
 
     # MLX / safetensors case: look for the dedicated shard or any
     # ``mtp_*`` key in the index.
-    model_dir = p if p.is_dir() else p.parent
+    model_dir = _snapshot_dir(p if p.is_dir() else p.parent)
     if (model_dir / "mtp.safetensors").exists():
         return True
     weight_map = _read_safetensors_index(model_dir)
+    if weight_map is None:
+        weight_map = _read_single_safetensors_keys(model_dir)
     if weight_map is None:
         return None
     for tensor_name in weight_map.keys():

backend_service/inference/_tensorfold.py

@@ -290,12 +290,27 @@ def match_model(
             return TensorFoldMatch(family=family, repo=repo, matched_by="repo")
 
     directory = model_dir or _directory_candidate(path, runtime_target)
-    family = _EXCLUSIVE_MODEL_TYPES.get(read_model_type(directory) or "")
+    model_type = read_model_type(directory) or ""
+    family = _EXCLUSIVE_MODEL_TYPES.get(model_type)
+    # "Exclusive" means stock MLX cannot load the architecture. mlx-lm 0.32 ships
+    # prism_hadamard_qwen35, so a small Ternary Bonsai 2 loads there, and
+    # TensorFold itself refuses every checkpoint but the 27B.
+    if family is not None and _stock_mlx_loads(model_type):
+        family = None
     if family is not None:
         return TensorFoldMatch(family=family, repo=None, matched_by="config")
     return None
 
 
+def _stock_mlx_loads(model_type: str) -> bool:
+    import importlib.util
+
+    try:
+        return importlib.util.find_spec(f"mlx_lm.models.{model_type}") is not None
+    except (ImportError, ValueError):
+        return False
+
+
 def _directory_candidate(*candidates: str | None) -> Path | None:
     for candidate in candidates:
         if not candidate: