Fix/benchmark broker and viz#275
Merged
Merged
Conversation
…es + no key nag Smoke-testing slopcodebench end-to-end surfaced three issues; all fixed in scb-visualize `run` (applied in the runner PTY so they can't be skipped and use THIS session's broker) + the template guide: 1. OpenRouter never reached the model. providers.yaml pointed opencode straight at openrouter.ai with the broker PLACEHOLDER key -> auth fail (total_cost=0, "OpenCode error"). Now prepare_configs() rewrites the openrouter openai api_base to $OPENROUTER_BASE_URL (the session's 127.0.0.1 broker, which injects the real key server-side). Idempotent across runs in the shared /workspace via regex. 2. The live agent viewer never surfaced. The tmux executor wrote runs.jsonl + the per-run logfiles into the run's temp working dir (cleaned up), not the checkout .scb_tmux that scb-visualize watches. prepare_configs() pins tmux_log_dir to <checkout>/.scb_tmux so they persist and are found. 3. Orchestrator nagged for OPENROUTER_API_KEY though it was already set (no tool to list secrets). Guide now says: don't ask up front, proceed, and only ask if the run errors with an auth/credit failure. Step 3 (manual providers.yaml patch) is gone — it's automatic now. Validated live (localhost + Playwright + Docker sandbox): providers.yaml rewritten to the broker URL, runs.jsonl/logfile persist in the checkout, the agent viewer appears on the canvas, and the run reaches the model (opencode/kimi-k2.6 making glob/read tool calls, net=$0.01288 real spend vs $0.00 before). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
A multi-problem run surfaces one read-only PTY per problem, which clutters the canvas. Add --no-viewers (and SCB_VIEWERS=0) to skip the per-run terminals and rely on the scb-live browser dashboard as the low-noise overview. Default stays on (current behavior); the run + dashboard are unaffected by the flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…run + scb-live) The template bypassed the fork's purpose-built tooling by calling slop-code directly, so the results browser was blank and the broker had to be re-patched by scb-visualize. Rewire step 4 to the intended pipeline and fix the browser: - Run via bin/scb-run (restores a pristine providers.yaml.orig, rewrites the openrouter api_base to $OPENROUTER_BASE_URL, launches the harness detached to .scb-run.log). - Start bin/scb-live and point the results browser block at :8051 (was :8050, the Dash port that needs evaluated runs) — it renders a live per-problem tally (checkpoint / steps / cost / state + cumulative cost). - bin/scb-visualize watch --skip-existing surfaces the optional per-problem viewers. - Setup now saves configs/providers.yaml.orig so scb-run can re-patch a clean copy each run (the per-session broker URL differs per run). - scb-visualize: add --skip-existing to watch mode (standalone use in the pipeline). Validated live (localhost + Playwright + Docker sandbox): scb-run patched the broker, scb-live served :8051 (HTTP 200, live table), the browser component rendered the dashboard, the per-run viewer surfaced, and the run reached the model (cost=0.01015). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…, workers) Phase 1 of "let the user configure the benchmark". A run is a matrix of (harness x model) arms, each running all `problems`, `workers` arms in parallel. - scb-matrix (new): reads /workspace/.scb-config.yaml (harnesses, models, problems, workers, prompt, thinking, evaluate; created with defaults if missing), or CLI overrides for a one-off (--harness/--model/--problem repeatable, --workers, ...). Expands to arms, patches the broker, runs W-parallel, tee'ing each arm to .scb-run.<arm>.<ts>.log with tmux mirroring (per-arm viewer). Dependency-free. - scb-live: matrix-aware — aggregates all logs in the latest batch into a per-ARM tally (arm | problem | checkpoint | steps | cost | state) + cumulative cost. - template: setup seeds the config; step 4 runs scb-matrix (+ scb-live + scb-visualize watch); guide teaches chat to edit the config or pass overrides. Both the chat and (later) a config panel edit the same .scb-config.yaml source of truth. Validated live (localhost + Playwright + Docker): "compare kimi-k2.6 and glm-4.6 on file_backup, 2 workers" -> chat built `scb-matrix --model … --model … --workers 2`; 2 arms launched via the broker, 2 per-arm logs, 2 viewers, and the results browser showed both arms with per-arm cost. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Control panel — a new "benchmark" canvas block (BenchmarkBlock) with dropdowns/ multi-selects for harnesses, models (OpenRouter catalog autocomplete), skills, problems, workers, thinking, prompt, evaluate, and a Run button. It reads/writes the SAME /workspace/.scb-config.yaml the chat + scb-matrix use (single source of truth): syncs from the file on mount, writes it on Run, then creates the runner terminal with the scb-matrix pipeline command. Wiring: DashboardItemType + BenchmarkContent, Canvas nodeTypes + sessionId injection + onCreateTerminalBlock callback, page blockTools/ defaultSizes/handleCreateTerminalBlock, files.ts read/writeSessionFile (PUT), and the controlplane VALID_ITEM_TYPES allowlist. Template ships a config panel item. Skills — a run is now a matrix of (harness x model x skill). A "skill" is a public agent-skill pack (baseline/gsd/omc/superpowers/karpathy) that swaps slop-code's --environment + --prompt (per the fork's REPRODUCE_PUBLIC_SKILLS.md). scb-matrix maps each skill to its (env, prompt), stages the skill plugin (clones the public repo, sets SCB_SKILL_<NAME>_DIR), and names per-arm logs with the skill. baseline + gsd run in the sandbox VM (local envs); omc/superpowers/karpathy have Docker-only envs (panel marks them ⚙ — need Docker-in-sandbox). scb-live's arm label includes the skill. Verified live: the panel renders on the template dashboard (harness pills, models/ problems chips, workers/thinking/prompt, and the Skills selector with local vs ⚙ Docker skills); config parser/expansion unit-tested (1h x 2m x 2s = 4 arms with the right env/prompt per skill); frontend + controlplane typecheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…trix - addyosmani: bring in the new public-skill from the fork's reproduce-public-skills branch (Docker env docker-python3.12-uv-with-addyosmani.yaml + trigger prompt). Added to scb-matrix SKILL_ENV/SKILL_PROMPT and the panel Skills list (⚙ Docker). - Broker patch robustness: scb-matrix restored providers.yaml.orig then only replaced the openrouter.ai URL — so a STALE broker URL baked into a non-pristine .orig (from a prior session) survived, and opencode hit a dead broker path (401 -> cost=0.0). Now also regex-rewrite any 127.0.0.1:PORT/broker/<sid>/openrouter to THIS session's broker (mirrors scb-visualize). Verified: after the fix the baseline arm bills (cost=0.175) via the current session's broker. Smoke test (panel-driven, localhost + Playwright + Docker): selecting baseline+gsd in the Benchmark panel and hitting Run wrote .scb-config.yaml, launched 2 arms with the correct per-skill env+prompt (baseline: local-tmux-py/just-solve; gsd: local-tmux-py- with-gsd/gsd-local-trigger), cloned the GSD plugin, and scb-live showed both arms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…ern) Previously only baseline + gsd had local (in-VM) slop-code envs; omc, superpowers, karpathy, addyosmani were Docker-only (bind-mount the skill from a host path), so they couldn't run in the Orcabot sandbox. Generalize GSD's local pattern to all of them: - scb-matrix SKILL_ENV/SKILL_PROMPT now point every skill at a local env (local-tmux-py-with-<skill>.yaml) + a `-local-trigger` prompt (relative .claude/skills/<dir> path instead of the Docker /tmp/agent_home mount). - SKILL_PLUGIN gains all 5 skills as (repo, subpath, envvar). stage_skill clones the public plugin once and points SCB_SKILL_<NAME>_DIR at repo/<subpath> — so a plugin that lives in a repo subdir (karpathy: skills/karpathy-guidelines) stages exactly that dir. The env yaml's setup `cp -R`s it into the run's .claude/skills/. - Panel: drop the ⚙ Docker marker + LOCAL_SKILLS gating — all skills run in the VM. - template.json guide + comments updated to match. The env yamls + `-local-trigger` prompts live in the fork (cloned at setup). Smoke test — all 5 skills validated two ways: 1. Deterministic (no spend): clone → subpath resolve → cp into .claude/skills/<dir> → each local trigger's referenced entry file resolves. Uses the real SKILL_* maps from this file. All 5 PASS (incl. karpathy's nested subpath). 2. Live billing (panel → broker PTY → opencode, ~$0.16 total): every skill clones, stages, and its local trigger drives opencode to the model — baseline $0.059, superpowers $0.032, karpathy $0.048 (subpath), omc $0.009, addyosmani $0.019 (fresh clone of addyosmani/agent-skills). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Adds a "subscription" auth path for Codex arms so they run on a ChatGPT/Codex
plan instead of a brokered API key. Orcabot's broker is API-key-only, but slop-code's
codex agent already supports a FILE credential (auth.json); this wires that up for
the in-VM local runtime.
scb-matrix: new `codex_auth: broker|subscription` config (+ `--codex-auth` /
`--subscription` flags). For a codex-subscription arm it (1) stages the located
auth.json into $AGENT_HOME/.codex/auth.json (searches $SCB_CODEX_AUTH_FILE,
$WS/.scb-codex-auth.json, ~/.codex/auth.json), (2) flips the openai provider in
providers.yaml from env_var to a file credential pointing at it, and (3) strips the
brokered OPENAI_*/OPENROUTER_* env so the codex CLI uses the subscription, not a
placeholder key. Model is passed as openai/gpt-5.5-codex (slop-code {provider}/{model}).
Panel: a "Codex: use my subscription" checkbox (shown only when codex is selected,
flagged ⚠ that the credential lives in the VM). BenchmarkContent gains codexAuth.
Validated in-VM with a dummy auth.json: staging + provider flip + env-strip + model
parsing all work; codex launches with CODEX_HOME=/tmp/agent_home/.codex, reads the
staged token, and reaches the subscription-auth handshake (rejected the dummy only on
`missing field id_token` — a real `codex login` auth.json has it). Live billing on a
real subscription credential is gated on user consent (credential exposure in the VM).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Adds the SlopCodeBench Runner to starter-templates.json so it's seeded into every fresh desktop DB like the other starters, instead of being DB-only (which meant a rebuilt/reset local DB lost it). Also wires setup_guide into seedStarterTemplates' INSERT — the SlopCodeBench template relies on its setup_guide (the chat-orchestrator walkthrough), which the starter seeder previously dropped. Bumps the seed marker v2 -> v3 so existing desktop DBs re-seed and pick it up on next /init-db (existing starters are ON CONFLICT(id) DO NOTHING, so it's purely additive). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Two setup papercuts: 1. No preselected problem. `problems` defaulted to [file_backup] in scb-matrix, the panel, and the template, so hitting Run could silently spend money on a benchmark the user never chose. Now it defaults to empty everywhere and scb-matrix exits(2) with a clear message listing the known problems instead of guessing. The panel's Run stays disabled until one is picked, and the setup guide tells chat to ASK which problem to run. 2. uv re-downloaded on every setup. The boot command ran `command -v uv || install`, only adding $HOME/.local/bin to PATH *after* installing — so the check always missed and re-downloaded, even though the binary persists at /workspace/.home/.local/bin/uv. Export PATH before the check. (Note: uv IS baked into the current Dockerfile at /usr/local/bin/uv as of 972e443, but images built before that — like the one currently shipping — don't have it, so the fallback still matters.) starter-templates.json's copy is regenerated from template.json so the shipped starter and the benchmarks/ source stay identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
1. Empty `problems` now means RUN THE FULL SET, not an error (corrects the previous
commit, which wrongly made it exit(2)). slop-code already auto-discovers every
valid problem when no --problem is passed (_resolve_problem_names -> discovery),
so scb-matrix simply omits the flags and logs "ALL problem(s)". The panel no
longer blocks Run on an empty problem list and shows "runs the full benchmark
set, one after another"; the arm summary reads "all problems". No single problem
is preselected anywhere.
2. list_secret_names chat tool — fixes chat nagging for a key that is already set
(it had no way to check, so it guessed). Guardrails:
- NAMES ONLY. Never selects `value`, not even a prefix — a "helpful" preview is
a real leak. Returns strictly set/not-set metadata (+type/broker_protected/
scope), so the broker guarantee (LLM never sees values) is preserved.
- owner/editor gate, matching run_command; viewers cannot enumerate.
- Scoped to the dashboard + '_global'.
This grants no new capability: brokered secret NAMES are already visible in the
PTY env as `NAME=[BROKERED]…` placeholders, and chat can already run `env` via
run_command — this just makes it reliable instead of a hacky shell call.
System prompt + the benchmark setup guide now require checking before asking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
The slop-code-bench fork work was done in an ephemeral scratch clone and exported here as a patch series + bundle so it wouldn't be lost (the scratch dir was indeed cleared). Those commits now live where they belong — on feat/host-tmux-executor in the fork checkout at ~/work/hyper/slop-code-bench — so duplicating fork code inside this repo is no longer needed.
The setup guard was `[ -e .../bin/scb-visualize ] || git clone …`. On a VM whose
/workspace/slop-code-bench predated that script, the guard fired, git refused to
clone into a non-empty directory ("fatal: destination path already exists"), and
setup fell through to print SETUP_OK anyway — leaving a stale checkout with no
bin/scb-matrix, permanently. Observed on a real VM (a Jul-5 copy, not even a git
repo, missing scb-matrix entirely), so the runner could never start.
Now: if it's a git repo -> fetch + reset --hard to the branch tip; otherwise move
any existing directory aside to <dir>.stale.<ts> (nothing is deleted; prior
outputs/ are preserved) and clone fresh. Idempotent — verified against a replica
of the broken VM state: first run repairs + preserves, second takes the
fetch/reset path with no duplicate stale dirs.
Also: clone/fetch are now --depth 1 (a full-history clone timed out at 2min in
testing; setup needs no history), and providers.yaml.orig is refreshed with cp -f
now that clone/reset guarantees a pristine providers.yaml — removing the
broker-URL staleness bug at its source rather than only patching around it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…iew race Setup no longer needs a PTY or the user. Chat now runs it with run_command (timeout_s=300), which blocks and returns an exit code, so it continues to the next step automatically instead of the old "watch the setup terminal for SETUP_OK, then say go" dance. The script is unchanged and idempotent; the guide tells chat how to read the result (SETUP_OK -> proceed same turn; timed_out -> re-call, uv resumes; egress hold -> ask user to Allow). create_terminal is now reserved for genuinely long-running things (the benchmark run), matching run_command's own guidance. Also removes the runbook note block (2 items now: benchmark panel + results browser, laid out side by side), and adds list_secret_names / run_command to the guide's tool list. Live-view race fix: the results browser showed about:blank because the block's Chromium launches once (observed: chromium 11:30:54, scb-live 11:35:56) and never retries, so its single navigation hit a dead port. scb-live now starts during SETUP so :8051 serves from the beginning, both start sites are guarded with pgrep (a blind second start just fails with EADDRINUSE), and the guide tells chat to point the user at the results browser + mention refresh if it's blank. Confirmed by the user: the URL was always fine — manually loading localhost:8051 worked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…run note
The pgrep guard I added never worked: `pgrep -f bin/scb-live` also matches the boot
command's OWN shell, whose command line contains that string. So it always reported
"already running", short-circuited, and scb-live was never started — port 8051 refused
and the results browser showed ERR_CONNECTION_REFUSED. Verified live in the VM: the
old guard claims running while no python3 bin/scb-live exists; probing the PORT
instead starts it and 8051 returns 200. Fixed in all three call sites (setup script,
run boot command, panel buildBootCommand).
scb-visualize no longer creates a note block per run — the terminal title
("agent: <problem>") already names it, and the note rendered empty. Grid reflows to
one terminal per slot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
The template shipped a results browser AND chat created a second one (create_browser) during setup — hence two blocks, neither connecting. A browser block navigates exactly once and never retries, so any block opened before scb-live has a run to show is stuck on ERR_CONNECTION_REFUSED permanently; the user confirmed both only worked after manually reloading post-launch. The template now ships no browser at all (just the benchmark panel), and the guide tells chat to create exactly ONE, at 127.0.0.1:8051, immediately AFTER launching a run — reusing an existing one rather than adding a second. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
… zooms canvas
The chat-created results browser landed with EMPTY content: create_browser is an
MCP-UI tool taking {dashboard_id, url, position, size}, and the guide called it with a
bogus `name` — an omitted/empty url silently yields a blank browser block (the
frontend stores `content: cmd.url` verbatim). Relying on the LLM to get that right is
also fragile, and it left panel-launched runs with no results view at all.
So the Benchmark panel's Run now opens it deterministically via the already-wired
onCreateBrowserBlock, at 127.0.0.1:8051, only once a run exists — a browser block
navigates exactly once and never retries, so opening it earlier parks it on
ERR_CONNECTION_REFUSED. handleCreateBrowserBlock now reuses an existing browser on the
same URL instead of stacking duplicates (the two-browser report). The guide documents
the real create_browser signature and notes the panel already handles it.
Also: Cmd/Ctrl +/- now zooms the canvas instead of the webview, Cmd+0 resets to 100%,
ignored while typing in an input/textarea.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Blocks were landing on top of each other. Three independent causes: 1. Stale state. computePlacement closes over `items` from React state, so blocks created back-to-back (scb-visualize surfacing several agent viewers, or chat creating a terminal then a browser) were all placed against the SAME pre-update list and collided. Placements are now reserved for 5s until the real item appears. 2. scb-visualize used its own fixed grid from a hardcoded origin, blind to existing canvas contents — so viewers overlapped the benchmark panel / runner terminal. It no longer sends a position; auto-placement handles it. 3. The search took the FIRST fit on a coarse newSize/2 grid, so it missed real gaps and fell through to "right of everything" more often than it should. findAvailableSpace is extracted to src/lib/canvas/placement.ts (it's pure geometry — it belongs in a testable module, not buried in a 4.4k-line page) and rewritten to pack row-major on a fine grid, with clearance only as a tie-break. Two earlier attempts were measurably worse and are documented in comments: scoring by clearance parked blocks in the middle of open space and stranded gaps; minimising bounding-box AREA growth always favoured extending the longer axis, producing a runaway single-row strip. The fallback now minimises how far you must zoom out (max(w/viewW, h/viewH)) instead. On the real dashboard sequence that turns a 2784x532 strip into a compact 1308x1396 layout. 8 vitest cases cover: no overlap, gap respected, roomier-side choice, N-in-a-row, minimal-growth fallback, empty canvas, sidebar inset, and a regression replaying the exact panel/runner/viewers/browser sequence that overlapped in D1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
An empty "Setup Terminal" kept appearing. Cause: sessions.item_id is NOT NULL REFERENCES dashboard_items — a sandbox session cannot exist without a canvas item. Having stripped every terminal from the template, a fresh dashboard had no sandbox, so run_command failed with "No sandbox is running for this dashboard yet. Start a terminal first." — and chat did exactly that, creating a useless empty PTY. Rather than add a schema change to security-sensitive session code, setup is now folded into the runner terminal's boot command (it is idempotent: fetch/reset, reuse the venv). The ONE terminal you already expect does its own setup, so there is no second component, and first-run dependency installation is visible instead of silent. The guide tells chat not to run setup separately and that "no sandbox yet" on a new dashboard is expected — launch the run, don't paper over it with an empty terminal. Results browser now opens at 1200x750 (150% of the stock 800x500) — the live tally is a wide table and was cramped. onCreateBrowserBlock takes an optional size override. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
All six findings verified against the code before fixing; all were real. P1 results browser raced scb-live. Folding setup into the run made this worse: the prewarmed Chromium navigated while the clone/venv was still going, and browser navigation is one-shot with URL dedup preventing a retry. scb-live now starts immediately after the checkout, BEFORE the slow venv sync, shrinking the window from minutes to roughly clone time. P1 per-run viewers were missing for every non-baseline skill. prepare_configs pinned tmux_log_dir only in local-tmux-py.yaml, so the five skill variants kept the default (a temp run dir); their runs.jsonl never appeared where watch() looks, so gsd/omc/ superpowers/karpathy/addyosmani surfaced no viewer at all. Now globs local-tmux-py*.yaml. P2 first-run config was not persisted: the panel wrote .scb-config.yaml only when a session already existed, which is never true on a new dashboard, so the launched config lived only in CLI flags while the documented "source of truth" file kept defaults. The boot command now writes the config with printf (no session needed) and scb-matrix reads the file — flags dropped entirely. P2 export erased benchmark config: scrubber.ts had no 'benchmark' case so it hit `default: return ''`. Root cause was that 'benchmark' was missing from DashboardItemType; added, plus a case preserving the (non-sensitive) run config. P2 toolbar-created benchmark still defaulted to problems:["file_backup"], contradicting the empty/full-suite default. P2 the v3 seed marker would have re-inserted the whole starter catalog, resurrecting starters a user deliberately deleted. Split into two markers: v2 still guards the original catalog, and newly-added starters seed under their own marker. Boot command verified with bash -n and by printing the generated shell. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
… skill fix P1 results-browser race: the earlier reorder only shrank the window (scb-live still starts after fetch/clone) while the browser was created immediately, so a cold first run could still land on ERR_CONNECTION_REFUSED with no retry. Replaced hope with a handshake: the run clears /workspace/.scb-live.ready at the start and writes it from a background loop once :8051 actually answers; the panel polls for that marker via the session file API and only then opens the browser (giving up after ~5min with a message pointing at the runner terminal). P1 skill-viewer fix was unreachable: prepare_configs() only runs in `scb-visualize run` mode, but the benchmark launches `watch`. Baseline worked only because local-tmux-py.yaml carries tmux_log_dir statically. Pinned it statically in all five local-tmux-py-with-*.yaml too, so the fix no longer depends on a code path the benchmark never takes. (The glob patch stays for `run` mode.) P2 removed committed Python bytecode (bin/__pycache__/*.pyc) and added __pycache__/ + *.pyc to .gitignore — they were re-added by an earlier `git add -A`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…owser
The readiness handshake was only wired into the Benchmark panel. The setup guide
still told chat to create the results browser "immediately AFTER" launching a run,
so chat-launched runs kept the original race: a browser block navigates once and
never retries, so one created while the harness is still cloning/building stays on
ERR_CONNECTION_REFUSED forever.
The guide now has chat read_file(".scb-live.ready") first and only create the browser
once that succeeds; if it errors, say the run is still preparing and re-check on the
next message. Also states explicitly that chat cannot poll in the background, so it
must not claim to be monitoring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…scription) The auth choice was effectively invisible. Codex-subscription mode existed only as a checkbox rendered when `codex` was already in Harnesses — and the default harness is opencode, so it was never shown. OpenRouter was never an option at all; it was implied by a model name starting with "openrouter/". Adds a Provider row at the top of the panel with two explicit choices: - OpenRouter (brokered key) — routed through the session secrets broker - Codex subscription — ~/.codex/auth.json, no API key Because the choice constrains both harness and model, selecting it fixes them up: subscription pins the harness to codex and swaps an openrouter/* model for openai/gpt-5.5-codex; switching back restores the OpenRouter default. Model suggestions and the placeholder follow the selected provider, and subscription mode shows where auth.json must be staged plus the warning that the credential is readable by the agent-under-test inside the VM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…reflight + Stop Root cause of the failed subscription run: the model, not the auth. The VM had a valid ChatGPT login all along (auth_mode=chatgpt, id_token+access_token present, and scb-matrix staged it correctly) — but I had picked openai/gpt-5.5-codex, and the "*-codex" variants are API-key only. codex rejected every instance with "The 'gpt-5.5-codex' model is not supported when using Codex with a ChatGPT account". The subscription-capable model is plain gpt-5.5, which is what the SWE-bench Pro reproduce-public-skills harness uses (MODEL="gpt-5.5"). Added configs/models/gpt-5.5.yaml and pointed the panel at openai/gpt-5.5. Because problems defaults to the FULL suite, that bad config didn't fail once — it failed identically for every problem while scb-matrix kept going, spawning a dead viewer terminal per run. Added a preflight that refuses to start a subscription run with no auth.json or a '*-codex' model, so this class of misconfiguration can't stampede. Stop button: writes /workspace/.scb-stop, which scb-matrix checks before each arm. Honest limitation — it stops launching further work, it can't interrupt an agent mid-problem. Guide: pick the provider BEFORE any credential check. Chat was asking about OPENROUTER_API_KEY even in subscription mode, where no API key is used at all. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Correcting an overreach in the previous commit. From a single error message I claimed all "*-codex" model variants are API-key only and hard-blocked them. That's not supported by the evidence: Codex CLI on a ChatGPT subscription demonstrably works, and which model ids a given plan exposes is account-dependent. The defensible fact is narrower: "gpt-5.5-codex" was an id I invented (mirroring gpt-5.3-codex from the fork's configs) and this account rejected it; gpt-5.5 is what the SWE-bench Pro harness drives. So: no model-id blocklist. Preflight now only checks the unambiguous case (subscription with no auth.json). The stampede is handled where it belongs — a watchdog that tails an arm's log and aborts when the failure will repeat for every problem (unsupported model, 401, bad auth, exhausted quota), setting the stop flag. Verified: injecting the real error message into a live arm's log terminates the process and writes the flag. This also covers failure modes a model whitelist never would. Panel/guide reworded to present gpt-5.5 as a known-good default, not a rule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
179 chars against 30-41 for the other starters, so it dominated the New Dashboard grid. It was also stale: it promised "One-time VM setup required (fork + venv)", which stopped being true once setup was folded into the run. Now "Compare coding agents on slop-code-bench" (39), matching the house style. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…ding it as failure Verified a live subscription run works: codex authenticated, reached the model and did real work on cfgpipe (py_compile, venv creation, schema fixtures, 12+ tool calls), with no auth or model errors. But the dashboard showed total \$0.0000 / steps 0/0 — which is exactly the signature of the earlier never-reached-the-model failures. Cause: codex on a ChatGPT/Codex plan emits no token_count/total_cost events at all, so cost is legitimately \$0 forever. Only broker/API-key runs bill per token. Our "success = cost>0" heuristic was therefore wrong for subscription runs and would have had chat reporting a healthy run as broken. scb-live now renders "plan" instead of \$0.0000 when codex_auth=subscription, and the guide judges success by checkpoint progression + real tool calls in the agent viewer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…ee staging Four review findings, all reproduced before fixing. P1 a failed matrix exited 0. run_arm discarded rc and swallowed launch exceptions, so main always printed "matrix complete" — a matrix where nothing ran was indistinguishable from success. Arms now return a status, main lists per-arm failures and exits 1, and the boot command surfaces it as "[matrix done rc=$rc]". Verified: a doomed launch now exits 1. P1 one bad arm cancelled the whole matrix. The fatal watchdog wrote the GLOBAL stop flag, terminating every other watcher and skipping queued arms — so a model-specific failure killed the comparison arms it existed to be compared against. Fatal state is now per-arm; only the user's Stop button writes the global flag. Verified: a fatal arm terminates itself and leaves the flag unset. P2 Run re-enabled immediately, allowing a competing runner against the same checkout, config, stop flag and timestamped logs. The run now writes /workspace/.scb-running (trapped so it clears on kill) and the panel polls it, showing "Running…" and keeping Run disabled for the actual duration rather than for the second it took to create a terminal. P2 parallel skill arms raced on clone. Workers each tested-and-cloned the same dest with rc ignored and success logged unconditionally, so a worker could see a partial checkout or silently run with no skill. Skills are staged once up front, cloned via temp dir + atomic rename, rc and non-empty plugin dir validated; an arm whose skill failed to stage is failed rather than quietly running without it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
P1 the guard was neither early nor atomic. It was created AFTER clone/venv/setup — minutes on a first run, during which Run stayed enabled — and used `: >`, which never rejects an existing file. Two clicks (or a click plus a chat-launched run) both entered the same checkout. The lock is now the FIRST thing the boot command does, taken with `set -o noclobber` so creation is atomic, and a losing run aborts instead of proceeding. Verified: concurrent starts -> second refused with "a benchmark run is already active". P2 a crash could disable Run permanently. Traps don't run after SIGKILL/VM crash/power loss, so a stale lock would have disabled the button forever with no recovery control. The lock now carries pid + a 20s heartbeat: the boot command reclaims it if the recorded pid is gone, and the panel treats a lock with no heartbeat in 90s as stale. Verified: stale lock (dead pid) is reclaimed. P2 partial skill checkouts were still accepted. Any non-empty plugin dir counted as staged, so an interrupted clone containing only .git passed — including legacy dirs left by the earlier race — and the arm ran without its skill. A completion marker is written inside the temp dir only after the clone is validated and published by the atomic rename; anything lacking it is re-cloned. Verified with a .git-only checkout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…reclaim P1 a successful run never released the lock. Cleanup relied on an EXIT trap, but the command ends in `exec bash` — exec replaces the shell so the trap never fires, and the heartbeat subshell keeps running under the same PID, refreshing the lock forever. Every completed benchmark therefore left Run disabled until the terminal was killed. The heartbeat is now killed and the lock removed explicitly before exec. Verified: lock is gone after a normal completion and a later run acquires cleanly. P1 `kill -0` cannot judge liveness across the PTY UID pool. Each PTY runs under its own UID, so probing a live runner owned by another PTY returns EPERM — which the previous check read as "pid is gone", deleted a LIVE lock, and re-opened the concurrent-run corruption it was added to prevent. Reclamation is now based purely on heartbeat staleness (>90s), which needs no cross-UID signal. Verified against a lock owned by an unprobeable pid with a fresh heartbeat: refused rather than stolen; a stale lock is still reclaimed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…mis-config) Live in the GUI, chat honoured "use codex subscription, baseline" by writing a config with harness=opencode + model=openai/gpt-5.5-codex + codex_auth=subscription, then reported "Launched for Codex". opencode can't use the codex CLI login (~/.codex/auth.json), so every problem failed rc=1. The new honest exit code surfaced it correctly ([matrix done rc=1], "matrix finished with FAILURES: 0 ok, 1 failed") — that part worked. Two fixes so the mis-config can't happen / is caught: - scb-matrix: the subscription preflight now fires whenever codex_auth=subscription (not only when 'codex' is already a harness) and refuses any non-codex harness with a message pointing at harness=codex + openai/gpt-5.5. - Guide: codex-subscription mode must set harnesses:[codex], models:[openai/gpt-5.5]. The panel's Provider=Codex subscription already pins these; the guide now makes chat do the same instead of leaving harness at the opencode default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…ct state The results browser never appeared even though .scb-live.ready was written at 83s (well inside the panel's 300s poll window) and was readable via the file API (200). Root cause: the panel opened the browser from component state — set awaitingLive on Run, poll .scb-live.ready, then create_browser. But launching the run adds the runner terminal to the canvas, which re-renders/re-mounts the benchmark block and resets awaitingLive to false, so the poll was dead before the marker appeared. The per-run agent VIEWERS, created server-side by scb-visualize via MCP, appeared fine — same GUI path, no fragile client state. So the fix is to open the results browser the same way: - scb-visualize waits for scb-live's port to answer, then create_browser once (marker-guarded so re-launches don't stack). Robust to any panel re-render. - Panel no longer polls/opens the browser (removed awaitingLive + effect). Kept the separate runActive poll that guards the Run/Stop button. - Guide: browser auto-opens; chat must NOT create_browser (would duplicate). This is the "eliminated the race too hard" fix: we'd swung from create-immediately (connection-refused race) to a client-state poll that gets wiped. Server-side MCP creation after readiness is both race-free and re-render-proof. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Ships the SlopCodeBench Runner as a baked-in starter template (seeded on init via seed_desktop_slopcodebench_v1) plus the full benchmark-runner surface: config panel with provider selector + Stop, flock run-lock, server-side results-browser open, codex-subscription auth, and the honest-exit-code / arm-isolation / race-free-staging fixes from this branch. Version bumped in tauri.conf.json, Cargo.toml, Cargo.lock; release cut separately via desktop/scripts/release.sh (signed + notarized). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
… run-lock The SlopCodeBench run-lock uses flock(1) for atomic, crash-safe, UID-pool-safe mutual exclusion. flock ships with util-linux; make it an explicit dependency rather than relying on it being pulled in transitively, so a future base-image change can't silently drop it and break the lock.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.