Skip to content

Provider registry: subscription-auth harness backends; cursor-agent Strict backend (M1) - #81

Merged
tzhouam merged 2 commits into
mainfrom
providers/registry-and-cursor-backend
Aug 14, 2026
Merged

tzhouam merged 2 commits into
mainfrom
providers/registry-and-cursor-backend

Conversation

@tzhouam

@tzhouam tzhouam commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Part of #80 (RFC — provider registry). Implements M1: the unified provider registry, the MCP tool bridge, and the cursor-agent harness backend. claude-code (M2) and codex (M3) are declared in the registry but unshipped — selecting them reports the milestone in strict_readiness/doctor instead of failing mid-run.

What this does

Strict previously required a raw Anthropic/OpenAI API key. With this PR, .env selects the execution backend explicitly:

STRICT_BACKEND=api        # today's raw-API path (unchanged)
STRICT_BACKEND=cursor     # Strict on a Cursor subscription via cursor-agent

Selection is explicit by design: a Strict run with STRICT_BACKEND unset refuses upfront with the exact fix line (same philosophy as TierNotConfiguredError); the CLI path defaults to api so existing setups keep working. .env.template now writes STRICT_BACKEND=api.

Design (details in #80 / doc/RFC-provider-registry.md)

  • One seam. ResolvedTarget gains provider_id/kind; tier_target stays the only resolution site. The only runtime branch is in run_agent_step: harness targets delegate the whole agent step (run_session) with the same prompt bundle, output contract, coercion and salvage; api targets are byte-identical to before (existing suite is the parity gate). MoA members keep the in-process loop — they carry their own API endpoints.
  • Choke point preserved. tool_bridge.py is an stdio MCP server the harness session launches; every builtin tool call passes tools.dispatch with the step's deserialized ToolScope. Reads are additionally confined to the PR-time worktree + run dir (ToolScope path-guards only writes) — the .env-exfiltration guard. Bridge tool events land in bridge_trace.jsonl.
  • Detective layer for cursor. cursor-agent cannot fully disable built-in tools, so every session is post-audited (productized from eval/dataset/run_cursor_arm.py, which caught a real out-of-bounds read): containment violations and read-only write attempts are traced (harness_session event) and surfaced as refusals.
  • Sanitized env. Harness subprocesses get an allowlist env — vendor CLIs must never inherit ANTHROPIC_BASE_URL/API keys (a gateway-pointed base URL would misroute subscription calls).
  • Honest accounting. Usage parsed best-effort from the CLI stream; no fabricated USD. HarnessLLM serves tool-less create() calls (reducer, repair, intent) via one-shot CLI invocations in an empty scratch cwd, and raises loudly if anything passes tools.

Known M1 gaps (disclosed in the RFC)

  • In-process extra tools built from a live StepContext (skill/memory retrieval, repo_map) are not reconstructed in the bridge — harness sessions get builtins + doc_search/doc_read.
  • Whether cursor-agent can disable built-ins with MCP tools configured is the first live-smoke question; the audit is the enforcement layer either way.
  • A live Strict smoke on a real PR with a logged-in cursor-agent is still to be run; all coverage in this PR is offline (fake CLI fixture).

Verification

  • Full suite green: 45 test files including 25 new tests (test_providers.py, test_tool_bridge.py, test_provider_cursor.py, extended test_mcp.py readiness tests).
  • ruff check clean on all touched files; git diff --check clean.
  • STRICT_BACKEND=api parity: zero changes to llm.py/agent_loop.py; existing tests unchanged except the three strict_readiness tests that now set the backend explicitly (the designed migration).

🤖 Generated with Claude Code

tzhouam and others added 2 commits August 14, 2026 17:29
…sor-agent Strict backend (M1)

Strict previously required a raw Anthropic/OpenAI API key. This lands the
provider registry (doc/RFC-provider-registry.md): one table of ways to
reach a model, with the existing raw-API path unified under it as provider
"api" (byte-identical parity, existing suite is the gate) and harness-kind
providers for subscription-authed vendor CLIs. cursor-agent ships as the
first harness backend; claude-code/codex are declared and ship in M2/M3.

- providers/: ProviderSpec + Transport contracts, registry, HarnessLLM
  (tool-less create() adapter; raises on tools), cursor transport
  (stream-json parse, sanitized allowlist env, scratch-cwd one-shots),
  productized session audit (from the Composer eval arm)
- tool_bridge.py: stdio MCP server exposing the run's ToolScope-checked
  tools to harness sessions via tools.dispatch, plus read containment
  (reads confined to worktree+run dir — ToolScope path-guards only writes)
  and a separate bridge_trace.jsonl
- config: STRICT_BACKEND selection (explicit; Strict refuses when unset,
  CLI defaults to api), ResolvedTarget gains provider_id/kind, tier_target
  returns harness targets with no endpoint/credential
- run_agent_step branches on target.kind: harness steps delegate the whole
  step (same prompt bundle, output contract, coercion, salvage); MoA
  members keep the in-process loop
- strict_readiness + doctor report backend selection/CLI gaps with exact
  fixes; .env.template documents the new keys

Offline coverage: registry selection, harness target shape, HarnessLLM
guards, bridge spec round-trip + containment refusals, cursor transport
against a fake CLI (session, audit flags, timeout, one-shot, env
sanitization).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…it widened to all path-bearing tools

Findings from the first live Strict runs on a real Cursor subscription
(PR #81 e2e, three runs on vllm-omni#6199):

- headless cursor-agent does not auto-approve configured MCP servers; the
  tool bridge was silently ignored until --approve-mcps was passed. With
  the flag, bridged grep/read_file calls flow through tools.dispatch and
  land in bridge_trace.jsonl (verified live: 26 and 8 bridge events).
- live usage keys are camelCase (inputTokens/outputTokens); the parser
  accepted only snake_case and recorded zeros.
- the audit checked containment only on readToolCall; native grepToolCall
  (and friends) carry paths too and sailed past a read-only audit — every
  path-bearing native call is now checked, and the misleading
  bridge_calls trace field is renamed other_tool_calls.
- cursor-agent spools MCP tool RESULTS under ~/.cursor/projects/*/
  agent-tools/ and reads them back; that read-back is bridge-output
  consumption, now exempt from root containment instead of flagging
  every bridge-using session.

Final verification run: audit_ok=true, zero violations, bridge engaged,
review success on the PR-time tree. Full offline suite green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tzhouam

tzhouam commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Live e2e: PR review in all three modes ✅

Ran the M1 acceptance smoke end-to-end on this branch — real MCP stdio sessions against the thin server, real PRs, real backends. All three modes pass; the cursor runs surfaced four real bugs, fixed in a8311cc.

Mode Target Result
Direct vllm-omni#6181 (head e85b2e4f) ✅ 7 tools listed; routed to the correct owner (components/diffusion/rules.md, quick_map ok, 3.1k chars); code-profile execution_budget with hard ceiling; validate_direct_review → complete / publish_ready. New gate verified: review(mode="strict") with STRICT_BACKEND unset refuses with the exact fix line.
Strict api (deepseek-v4-pro @ api.deepseek.com) vllm-omni#6181, light depth ✅ run_id → poll → paged report, done in 541s. PR-time checkout pinned; dispatch trace shows endpoint: api.deepseek.com, source: global. 22 tool calls, 142.8k in / 25.6k out. Review: 2 major findings (TeaCache do_true_cfg cache-contamination risk on the newly-enabled negative branch, with repo-wide grep evidence) → REQUEST CHANGES via the escalate-salvage path.
Strict cursor (Composer 2.5, subscription auth) vllm-omni#6199, light depth ✅ (3 runs; final fully green) dispatch trace model: composer-2.5, endpoint: cli:cursor, source: backend:cursor; MCP tool bridge engaged — bridged grep/read_file calls through tools.dispatch recorded in bridge_trace.jsonl (26 events run 2, 8 events run 3); harness_session audit_ok, zero violations, no timeout; usage captured (47.8k in / 6.2k out); review success, high confidence, 1 grounded comment (missing padded-input unit test, with file:line evidence and a runnable pytest suggestion).

Live findings → fixes (a8311cc)

  1. --approve-mcps is load-bearing. Headless cursor-agent silently ignores configured MCP servers without it — run 1 completed on native tools only, with the bridge spec written but never launched. This answers the RFC's M1 open question: built-ins cannot be disabled, so the shipped posture is bridge + native side-by-side with the detective audit, exactly the decided fallback.
  2. Usage keys are camelCase live (inputTokens/outputTokens); the parser read snake_case and recorded zeros.
  3. Audit widened to every path-bearing native tool. Only readToolCall was containment-checked; native grepToolCall etc. carry paths too and sailed past a read-only audit (they were also mislabeled bridge_calls in the trace — now other_tool_calls).
  4. MCP result spool exemption. cursor-agent buffers MCP tool results under ~/.cursor/projects/<session>/agent-tools/ and reads them back; the audit flagged every such read-back as out-of-tree. That pattern (and only that pattern) is now exempt.

Env notes: harness subprocess env sanitization verified live (ANTHROPIC_BASE_URL gateway never reached the CLI); both Strict runs pinned PR-time worktrees; the two backends ran concurrently on different PRs without contention.

Full offline suite green after fixes (26 provider/bridge/cursor tests). Run artifacts: run-20260814-173654-dc4193 (api), run-20260814-174749-e4589d (cursor, final).

🤖 Generated with Claude Code

@tzhouam
tzhouam merged commit 8354321 into main Aug 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant