Windows compatibility for launcher, CLI, and config paths#11
Open
offbyonebit wants to merge 14 commits into
Open
Windows compatibility for launcher, CLI, and config paths#11offbyonebit wants to merge 14 commits into
offbyonebit wants to merge 14 commits into
Conversation
- Split monolithic chat.html into chat.css + chat.js modules - Render assistant messages with marked.js + highlight.js syntax highlighting - Add copy buttons to code blocks - Redesign agent tool cards as a step timeline with step numbers and target chips - Render markdown in agent assistant messages - Improve assistant message typography (no global monospace)
Adds a Windows code path alongside the existing Linux one: process lifecycle (CREATE_NEW_PROCESS_GROUP + CTRL_BREAK_EVENT/taskkill tree-kill instead of setsid/killpg), %APPDATA%/%LOCALAPPDATA% config paths, and graceful degradation in `doctor`/`init`/`serve`/`systemd-unit` for Linux-only diagnostics. Also fixes 13 pre-existing ruff lint errors (unused imports/locals) that have been silently failing CI's lint step since the mtp-clean merge, so Test never actually ran. Adds windows-latest to the CI matrix to get real validation on a Windows runner.
The windows-latest CI job (added in the previous commit) caught 7 real
failures the Linux-only local verification couldn't:
- search_files now emits POSIX-style relative paths (Path.as_posix())
instead of native separators, so matches are consistent across platforms.
- test_run_command_cwd_is_root used `pwd`, which on the Windows runner
resolves to Git Bash's pwd.exe and prints an MSYS-style path
("/c/Users/...") instead of a native one even though cwd was correct.
Swapped for a python -c cwd check, which is honest on both platforms.
- make_sysfs_gpu fakes Linux's /sys/bus/pci/devices/<slot> layout, and
slot names contain colons (e.g. "0000:03:00.0"), which are illegal in
Windows path components. This fixture is inherently Linux-only (Windows
has no sysfs to fake), so it now skips on win32 instead of failing.
test_find_drm_card_found builds its own /sys/bus/pci/devices/0000:03:00.0 path inline instead of going through the make_sysfs_gpu fixture, so it slipped past the skip added there. Same root cause: colon in the PCI slot name is illegal in a Windows path component, and there's no real sysfs to fake on Windows anyway.
…t export/import Runtime robustness: - Router refuses to load a model if its estimated VRAM (weights + KV cache + overhead) won't fit on the target GPU, accounting for co-resident models in multi-resident mode. - LlamaServer.tail_log() surfaces the last 40 log lines in the error when a model fails its health check, instead of just a log line nobody sees. - load_config now runs migrate_config()/validate_config(): rejects configs newer than the supported schema version, fills in missing top-level sections, and raises clear errors on structural problems. Operations/observability: - GET /admin/metrics: load/stop/error counters, active models, GPU info. - GET /health now reports uptime and currently loaded models. - llama-server subprocess logs rotate at 50MB, keeping 3 backups. - ARC_LLAMA_LOG_JSON=1 switches logging to single-line JSON. API/UX: - GET /v1/chats/export and POST /v1/chats/import, plus Export/Import buttons in the chat history panel. Reuses the existing path-sanitizing _chat_path, so imported chat ids can't escape the chat directory. Streaming in the web UI and agent tool confirmation were already implemented prior to this change; no work was needed there.
Adds a Backend enum (sycl/vulkan) with per-GPU backend config, branching build_env() to strip SYCL/oneAPI vars and set GGML_VK_VISIBLE_DEVICES on the Vulkan path. New binary.py detects which backend a llama-server build actually supports (via strings or a pure-Python scan) and surfaces mismatches in `doctor`/`add`/`init`. Also expands filename-based kv_class detection to cover Qwen2.5, Qwen3 dense, Phi-4, Llama 3/4, and DeepSeek-R1 distills. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…fault-safe admin auth Interactive agent TUI, MCP client support, repo map/semantic index, and per-run file checkpoints with restore. Checkpoint store now enforces retention (max count, max age, optional total-size cap) so .arc_llama_checkpoints/ doesn't grow unbounded. Admin/agent HTTP surface now requires a bearer token: _require_admin gates every destructive admin route plus auto_confirm agent runs, closing the gap where a caller could set auto_confirm:true in the request body and skip tool confirmation entirely. admin_token is no longer opt-in -- load_config auto-generates and persists one the first time a config has none, ARC_LLAMA_ADMIN_TOKEN overrides without persisting, and `serve` takes --admin-token and prints the active token at startup. The bundled web UI bootstraps its own token from /admin/session-token, which is gated on the connection being loopback rather than on the token itself, so it stays zero-friction locally but refuses to hand out the secret to non-loopback callers. Also: Vulkan-adjacent config/CLI polish, dropped two stale roadmap items now superseded by shipped work, and a test-isolation fix (three server tests were implicitly reading the real ~/.config/arc-llama/config.toml via a bare create_app()).
The auto-recipe forced -ub 8 for every MTP model on the assumption it prevented SSM compute-buffer OOM. Live benchmarking on real hardware showed this made prompt-eval ~9.5x slower than doing nothing (45.6 vs 435.7 tok/s on Qwen3.6-27B-MTP) while the OOM risk was never actually confirmed. Removed the forced ubatch size from both add_local_model and register_discovered and the launcher fallback; re-benchmarked after the fix and prompt-eval recovered to ~360 tok/s with generation throughput unchanged, matching hand-tuned production numbers. Also fixes MoE detection silently failing on real Gemma-4-26B-A4B and Qwen3-Coder-30B-A3B GGUFs (expert_count metadata was never read), and benchmark.py not sending the admin bearer token after admin auth landed. Bumps to 0.4.0 for the accumulated Windows support, Vulkan backend, agent TUI/MCP/checkpoints, and default-safe admin auth since 0.3.0.
Refocuses the default CLI and web UI on the core goal: fastest way to set up and run inference on Intel Arc/Battlemage cards. The agent, code, and agent-tui commands are still there, but hidden from --help and the bundled chat UI unless ARC_LLAMA_EXPERIMENTAL_AGENT=1 is set. Drops the standalone `arcllama` script entry point since it's now redundant with `arc-llama agent-tui`, and fixes the README/cli.py docstring references that still pointed at it.
Launch tuning grounded in real B60 measurements rather than folklore: - policy: apply_launch_policy() centralises only verified adjustments (Vulkan quantized KV needs --flash-attn; warn, don't force, when draft-mtp spec_draft_n_max > 4). Drops the earlier unbenchmarked "SYCL MTP net-negative on Xe2" auto-switch and the SYCL hard-abort. - recipes: default_recipe bumps -ub 512->1024 on >=16 GB cards and budgets the larger compute buffer into ctx sizing. Measured on Qwen3.6-27B-MTP: ~340 -> ~420 tok/s prompt-eval (~23%, all runs non-overlapping over 5 repeats x2), no gen regression. Small cards stay at the stock ubatch so a fitting model doesn't OOM. - platform_checks: doctor host/driver checks (ReBAR aperture, render group, xe/i915 VRAM readers) that previously returned None on Intel. - scripts/bench_results: empirical_throughput.py harness plus the SUMMARY the ubatch and n_max conclusions are drawn from.
Land the remaining useful pieces from claude/intel-inference-performance onto windows-compat (ubatch default already cherry-picked in 9a9a106): - arc-llama tune: staged greedy sweep (KV → ubatch → flash-attn), winner persisted to the model recipe; dry-run restores the original - arc-llama benchmark: --json, repeatable --kv, admin-auth headers (CLI front-end was already present; polish + VRAM plumbing completed here) - recipes: flash_attn / batch_size / no_mmap / mlock fields, recipe_to_dict, PERF_BATCH paired with the measured ubatch default - server_caps: probe llama-server --help for -fa dialect (boolean vs on|off|auto) so old and new binaries both launch - VRAM: xe tile0/physical_vram_size_bytes + i915 lmem_total_bytes totals, per-process DRM fdinfo via backend pid from /admin/status (amdgpu-only readers previously returned None on every Intel card) - Dockerfile: GGML_SYCL_DEVICE_ARCH AOT build arg, GGML_SYCL_F16=ON, llama.cpp pin b9946, oneAPI 2025.3.2 Tests: 369 passed, 2 skipped.
…sed kv_class - arch.py: AOT_ARCH_BY_DEVICE_ID + aot_arch_for() computes the exact ocloc device string per detected GPU instead of leaving it as a Dockerfile comment; VRAM_BY_DEVICE_ID + known_vram_mib() gives detect.py a fallback when sysfs mem_info_vram_total is absent (no working clinfo needed). - cli.py doctor: prints the computed AOT build flag per GPU; serve now prints a startup banner (arch/backend/VRAM/ReBAR/JIT-or-AOT per GPU, ctx/KV/port per model) so the applied profile is visible instead of buried in code comments. - models.py: infer_kv_class_from_path()/resolve_kv_class() prefer GGUF metadata (expert_count, architecture) over the filename heuristic. Fixes a real misclassification on Gemma-4-26B-A4B (filename heuristic guessed dense gemma_swa; expert_count=128 correctly identifies MoE). - README: reconciled the Battlemage kernel requirement (was 6.8+ vs arch.py's 6.14+) to match what doctor actually warns on. Verified live on the B60: doctor prints the real AOT/VRAM lines, serve banner renders correctly, kv_class fix confirmed against production Qwen3.6-35B-A3B and Gemma-4-26B-A4B GGUFs. Full suite: 369 passed, 2 skipped.
Was hardcoded at "0.1.0" and never bumped alongside pyproject.toml, so every install (system pip 0.3.0, project venv 0.4.0) reported the same wrong version via arc-llama --version. Now reads importlib.metadata so it can't drift again.
Unused subprocess import and an unsorted import block in cli.py have been failing the Lint step on every CI job (both ubuntu-latest and windows-latest, all Python versions) since at least 2026-07-08 — Test and Build never even ran because Lint gates them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
%APPDATA%/%LOCALAPPDATA%), and CLI (doctor/init/serve/systemd-unitdegrade gracefully instead of crashing on Windows).windows-latestto the CI matrix for real validation on a Windows runner.Test plan
ruff check .passes locallymypyclean on changed files