Skip to content

[Feature] Add agent bridge and webui update#1364

Merged
Yunnglin merged 15 commits into
mainfrom
add/agent_bridge
May 25, 2026
Merged

[Feature] Add agent bridge and webui update#1364
Yunnglin merged 15 commits into
mainfrom
add/agent_bridge

Conversation

@Yunnglin

@Yunnglin Yunnglin commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Adds agent bridge: a process-internal protocol-translation proxy that lets external CLI coding agents (claude-code, codex) drive any evalscope-configured model on any benchmark. Plus a web-view refactor folded into the same branch.

The bridge speaks three wire protocols so a single evalscope Model can serve agents that expect very different APIs:

Route Protocol Verified client
POST /anthropic/v1/messages Anthropic Messages claude-code
POST /openai/v1/chat/completions OpenAI Chat Completions (generic OpenAI clients)
POST /openai/v1/responses OpenAI Responses (typed input[]) codex v0.133+

Each route translates the incoming request → ChatMessage[] → model.generate_async → reverse-translates ModelOutput → synthesises the wire-shape SSE stream back. The bridge is a protocol translator, not a passthrough proxy — downstream LLMs (e.g. DashScope qwen3-max, which only speaks chat completions) still work transparently for clients that demand Anthropic or Responses on the wire.

Why this design

Existing AgentLoop is great for evalscope-native evaluations, but production-grade coding agents (claude-code, codex, cursor-agent, …) are external CLI binaries with their own loop/strategy/tooling. Wrapping them as a Model adapter loses fidelity; the bridge instead lets the CLI run unchanged and intercepts at the HTTP layer, capturing the full trajectory in evalscope's existing AgentTrace schema so all downstream tooling (scoring, viewer, reports) works without modification.

What's in the PR

evalscope/agent/external/

  • bridge/ — aiohttp server (server.py), three pairs of translate_*.py + sse_*.py modules per wire protocol, shared _sse_common.py. trace_recorder.py emits AgentTrace/AgentTraceEvent shapes identical to native AgentLoop so downstream serialisers see a single trace format.
  • runners/ — AgentRunner ABC (base.py) + concrete runners:
    • claude_code.py → claude --print via ANTHROPIC_BASE_URL
    • codex.py → codex exec -c model_providers.evalscope.* (Responses-only since v0.133)
    • mock.py for unit tests
  • adapter.py — run_external_agent glue: starts bridge → spawns CLI → captures trace → produces TaskState.
  • config.py / ExternalAgentConfig — declarative kwargs passed via TaskConfig.agent_config = {'mode': 'external', 'framework': 'codex'|'claude-code', 'kwargs': {...}}.

Bench / model integration

  • AgentLoopAdapter._on_external_agent_inference dispatches to the runner registry by framework name.
  • LocalAgentEnvironment + EnclaveAgentEnvironment are reused unchanged (the bridge is environment-agnostic).
  • Runners auto-install missing CLIs in the per-instance container (apt + nodesource + npm) so SWE-bench Pro / sweap-images don't need to be pre-baked. Default kwargs={} works end-to-end.

Web view refactor

evalscope/web/: warm-cream light theme + per-theme score CSS vars + several view components simplified (c73fdc4, 4fdd7d4, 94f61c0, 373e48b). Independent of the bridge — bundled here because it touches the report viewer that's used to inspect bridge-produced traces.

Docs

docs/{en,zh}/user_guides/agent/ split into index.md (overview) + native.md (AgentLoop) + bridge.md (external CLI bridge). Plus cross-links from sandbox / parameters / visualization / READMEs.

Usage

from evalscope import run_task, TaskConfig

# Drive codex × any DashScope chat-completions model on SWE-bench Pro
run_task(TaskConfig(
    model='qwen3-max',
    api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
    api_key='...',
    eval_type='openai_api',
    datasets=['swe_bench_pro'],
    agent_config={
        'mode': 'external',
        'framework': 'codex',           # or 'claude-code'
        'kwargs': {},                   # defaults cover everything
        'timeout': 1500.0,
    },
    sandbox={'default_config': {'memory_limit': '8g', 'cpu_limit': 4}},
    limit=3,
))

For claude-code use framework='claude-code' — same shape, dial any model behind the Anthropic Messages wire.

Verification

  • Unit tests (tests/agent/external/): 51 mock cases covering all three wire protocols (request translation, SSE event ordering + sequence numbers, multi-turn, tool-use, instructions/system merging, malformed-JSON tool arguments, 401 auth path).
  • End-to-end real runs (tests/benchmark/test_agent_loop.py, opt-in via DASHSCOPE_API_KEY):
    • TestGSM8KExternalClaudeCode — claude-code × qwen3-max × GSM8K
    • TestSWEBenchProExternalClaudeCode — claude-code × qwen3-max × SWE-bench Pro
    • TestSWEBenchProExternalCodex — codex × qwen3-max × SWE-bench Pro
  • Lint clean (make lint).
  • Downstream: cross-loop shutdown noise tracked separately in modelscope/ms-enclave#12. Not blocking — purely cosmetic process-exit log lines.

Notes for reviewers

  • Bridge is request-scoped, in-process — no separate sidecar, no port conflicts (random port per evaluation, isolated per-trial token).
  • previous_response_id (Responses-side stateful sessions) is deliberately not supported; bridge logs WARN and processes the full input[] each turn. Stateless retry is more useful for evals than session continuity.
  • codex's chat-completions parallel_tool_calls=True SSE bug (delta fan-out by tool_calls[i].index broken) is documented as a landmine in codex.py; codex v0.133+ avoids it by always speaking Responses. Other OpenAI clients that hit the chat path may need User-Agent detection if they trip it.
  • Tool registry: @register_runner('<framework>'). Third-party runners drop into evalscope/agent/external/runners/ and register the same way new benchmarks do.

Test plan

  • Mock test suite (51 cases) green
  • TestSWEBenchProExternalCodex real-API run (limit=3) green
  • make lint clean
  • Reviewer to run pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings for non-regression smoke

Yunnglin added 9 commits May 21, 2026 12:51
@Yunnglin
feat(external-agent): P0 bridge for third-party CLI agents
320d203 
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.

* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
  AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
  gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
  lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
@Yunnglin
rewise p0
9ab2892
@Yunnglin
update p0
1648aa0
@Yunnglin
update p0
8ed0dc9
@Yunnglin
update registry
1ea9b2e
@Yunnglin
update p1
6cd98e3
@Yunnglin
update web view
373e48b
@Yunnglin
simplify web view
94f61c0
@Yunnglin
openai sse proxy bridge
eb63890
gemini-code-assist[bot]
gemini-code-assist Bot reviewed May 22, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive integration for external agents, featuring an HTTP reverse-proxy bridge that translates Anthropic and OpenAI protocols to EvalScope's internal model API. Key additions include the ClaudeCodeRunner for executing third-party CLIs in sandboxes, a unified AgentTrace format for both native and external runs, and a refactored registry system to support aliases. The web dashboard is also updated with new components for trajectory visualization. Feedback identifies a critical bug in the bridge server where asyncio.Lock is incorrectly shared across event loops, potentially causing runtime errors; a switch to threading.Lock is recommended. Other suggestions include implementing proper cleanup for temporary directories in the agent runner and using Pydantic validation aliases to prevent breaking changes caused by field renaming.

Comment thread evalscope/agent/external/bridge/server.py
Comment thread evalscope/agent/external/bridge/server.py Outdated
Comment thread evalscope/agent/external/bridge/server.py Outdated
Comment thread evalscope/agent/external/bridge/server.py Outdated
Comment thread evalscope/agent/external/runners/claude_code.py
Comment thread evalscope/api/agent/types.py
Yunnglin added 5 commits May 22, 2026 17:16
@Yunnglin
update webui refactor
4fdd7d2
@Yunnglin
refactor(web): warm-cream light theme + per-theme score vars
c73fdc4 
- Light theme reworked from cool-slate translation to warm-cream Console:
  surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
  (#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
  tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
  yellow mid-tones stay legible on cream without forking the brand HSL.
  scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
  switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
  CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
@Yunnglin
openai responses api bridge + codex runner
8bf9261 
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.

* translate_responses.py: full coverage of codex/OpenAI input items —
  message / function_call / function_call_output / reasoning /
  custom_tool_call(_output) / computer_call(_output) plus opaque
  placeholder rendering for built-in tool items (web_search_call,
  mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
  image_generation_call, local_shell_call). item_reference + unknown
  types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
  event sequence (created → in_progress → output_item.added/done × N
  → completed), strictly monotonic sequence_number, 32-char
  function_call_arguments chunking matching inspect_ai. Upstream error
  path emits OpenAI SDK ResponseErrorEvent shape (flat
  type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
  and _prepare_sse_response helpers extracted; anthropic 401 shape
  preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
  new messages_key parameter; _ingest_initial_user_message filters
  type=='message' to avoid mis-ingesting function_call_output as user
  prompts; _extract_responses_tool_results WARNs once when an output
  appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
  -c model_providers.evalscope.* config injection, npm install
  fallback for codex CLI. Setup + run + answer extraction via
  --output-last-message.

Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
@Yunnglin
update codex bridge
6a5e186
@Yunnglin
external-agent: review cleanup, trace fixes, agent docs reshuffle
cb4d63c 
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
  drop output-last-message kwarg (constant). Remaining knobs: model_name,
  extra_args, extra_config, home_override, auto_install, install_timeout_s,
  node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
  with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
  so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
  kept as the default sentinel for callers that already passed it.

Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
  responses-API turn (top-level instructions + every system/developer item
  + every user item, in document order), not just the first user message.
  Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
  and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
  already-recorded ChatMessageTool entries, replacing the tail-only
  shortcut. codex re-sends every (function_call, function_call_output)
  pair interleaved through input[], not clumped at the tail, so the old
  logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
  results as new events.

Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
  -> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
  empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
  (subsumed by qwen_via_claude_code + walking_skeleton). Rename
  test_responses_instructions_merge.py -> test_responses_input_assembly.py
  to match what it actually covers.

Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
  (overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
  (external CLI bridge: claude-code, codex). Cross-link from sandbox /
  parameters / visualization / READMEs / index.

Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
  run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
  with no in-tree references; their code paths are now fully covered by
  tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
  TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.

Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
  (modelscope/ms-enclave#12)
@Yunnglin
Yunnglin marked this pull request as ready for review May 25, 2026 07:26
Copilot AI review requested due to automatic review settings May 25, 2026 07:26
Copilot AI reviewed May 25, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an external-agent bridge path alongside the native AgentLoop, updates agent configuration plumbing, expands bridge/test coverage, and refreshes the Web UI design system and documentation.

Changes:

  • Added external agent runner/bridge abstractions, config union support, runner tests, and SWE-bench patch extraction hooks.
  • Refactored native agent execution into a shared runner and renamed agent free-form options toward kwargs.
  • Updated Web UI tokens/components, report/task pages, trace event typing, and agent documentation.

Reviewed changes

Copilot reviewed 145 out of 146 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tests/agent/test_t2_environment.py Updates native agent tests for async generation and kwargs.
tests/agent/test_agent_interfaces.py Updates config default assertion to kwargs.
tests/agent/external/test_walking_skeleton.py Adds bridge smoke and auth tests.
tests/agent/external/test_responses_previous_response_id_ignored.py Adds Responses warning-path test.
tests/agent/external/test_responses_multi_turn.py Adds Responses multi-turn trace test.
tests/agent/external/test_responses_input_assembly.py Adds Responses translator ordering tests.
tests/agent/external/test_responses_error_paths.py Adds Responses upstream error tests.
tests/agent/external/test_qwen_via_claude_code.py Adds opt-in real Claude Code/Qwen test.
tests/agent/external/test_openai_chat_tool_calls_streaming.py Adds OpenAI chat SSE tool-call tests.
tests/agent/external/test_openai_chat_multi_turn.py Adds OpenAI chat multi-turn trace test.
tests/agent/external/test_helpers_patch.py Adds patch extraction tests.
tests/agent/external/test_evaluator_pipeline.py Adds evaluator pipeline bridge test.
tests/agent/external/test_codex_runner.py Adds opt-in real Codex/Qwen test.
tests/agent/external/test_agent_loop_external.py Adds AgentLoopAdapter external dispatch tests.
requirements/framework.txt Adds aiohttp dependency.
README.md Documents bridge feature and agent guide links.
README_zh.md Documents bridge feature and Chinese guide links.
evalscope/web/src/utils/colorScale.ts Centralizes score color helpers.
evalscope/web/src/pages/ReportViewerPage.tsx Updates theme tokens.
evalscope/web/src/pages/ReportsPage.tsx Updates text-dim comments/tokens.
evalscope/web/src/pages/PerfTaskPage.tsx Changes perf task running-state handling.
evalscope/web/src/pages/EvalTaskPage.tsx Changes eval task running-state handling.
evalscope/web/src/pages/BenchmarksPage.tsx Updates token usage.
evalscope/web/src/i18n/translations.ts Adds run start/end labels.
evalscope/web/src/hooks/useCopy.ts Adds clipboard hook.
evalscope/web/src/contexts/ThemeContext.tsx Adds OS theme fallback.
evalscope/web/src/contexts/ReportsContext.tsx Adds report cache limiting/useRef loading.
evalscope/web/src/components/ui/Tabs.tsx Updates filled-state tokens.
evalscope/web/src/components/ui/Table.tsx Removes local score helper and updates table styling.
evalscope/web/src/components/ui/Select.tsx Adds design-system comments.
evalscope/web/src/components/ui/SearchInput.tsx Adds design-system comments.
evalscope/web/src/components/ui/ScoreChip.tsx Adds score chip atom.
evalscope/web/src/components/ui/ScoreBadge.tsx Adds score badge atom.
evalscope/web/src/components/ui/PathBar.tsx Adds path input atom.
evalscope/web/src/components/ui/ModelGroupHeader.tsx Adds collapsible group header atom.
evalscope/web/src/components/ui/KpiCard.tsx Adds KPI card atom.
evalscope/web/src/components/ui/formStyles.ts Adds shared form classes.
evalscope/web/src/components/ui/FormField.tsx Adds form field wrapper.
evalscope/web/src/components/ui/Eyebrow.tsx Adds eyebrow typography atom.
evalscope/web/src/components/ui/EvalRunCard.tsx Adds eval run card atom.
evalscope/web/src/components/ui/Collapsible.tsx Adds collapsible atom.
evalscope/web/src/components/ui/ChatBubble.tsx Adds role-based chat bubble atom.
evalscope/web/src/components/ui/Card.tsx Uses Eyebrow.
evalscope/web/src/components/ui/Button.tsx Updates primary button tokens.
evalscope/web/src/components/ui/Breadcrumb.tsx Adds design-system comment.
evalscope/web/src/components/ui/Badge.tsx Updates success token.
evalscope/web/src/components/single/ChatView.tsx Refactors legacy message rendering and trace props.
evalscope/web/src/components/single/chat/roleConfig.ts Maps chat roles to ChatBubble tokens.
evalscope/web/src/components/reports/ReportSummaryStats.tsx Uses centralized score colors and new typography.
evalscope/web/src/components/reports/ReportHeader.tsx Uses centralized score colors and token comments.
evalscope/web/src/components/reports/ReportFilters.tsx Adds design-system comments.
evalscope/web/src/components/reports/ReportCard.tsx Uses centralized score colors and updated tokens.
evalscope/web/src/components/reports/OverviewTab.tsx Uses centralized score colors.
evalscope/web/src/components/reports/DetailsTab.tsx Uses centralized score colors and ring styling.
evalscope/web/src/components/reports/DatasetNav.tsx Updates typography token.
evalscope/web/src/components/nav/TopNav.tsx Updates navigation tokens.
evalscope/web/src/components/common/ScoreBadge.tsx Removes old score badge.
evalscope/web/src/components/common/Pagination.tsx Updates theme tokens.
evalscope/web/src/components/common/MarkdownRenderer.tsx Extracts lightbox and memoizes markdown components.
evalscope/web/src/components/common/LoadingSpinner.tsx Updates text token.
evalscope/web/src/components/common/ImageLightbox.tsx Adds shared image lightbox.
evalscope/web/src/components/common/FilterBar.tsx Updates surface/accent tokens.
evalscope/web/src/components/common/ErrorBoundary.tsx Converts inline styles to token classes.
evalscope/web/src/components/common/EmptyState.tsx Reworks empty-state component API/design.
evalscope/web/src/components/common/DataTable.tsx Updates table tokens.
evalscope/web/src/components/charts/PlotlyChart.tsx Updates label/error styling.
evalscope/web/src/api/types.ts Adds run start/end trace event types.
evalscope/web/src/api/reports.ts Simplifies report API parameter construction.
evalscope/web/README.md Documents Web design system atoms/tokens.
evalscope/web/index.html Applies OS theme before first paint.
evalscope/config.py Adds native/external agent config union validation.
evalscope/benchmarks/swe_bench/swe_bench_agentic_adapter.py Adds external patch extraction hook.
evalscope/benchmarks/swe_bench_pro/swe_bench_pro_agentic_adapter.py Adds external patch extraction hook.
evalscope/arguments.py Adds CLI --agent-config.
evalscope/api/benchmark/adapters/default_data_adapter.py Routes default adapter to native or external agent paths.
evalscope/api/benchmark/adapters/agent_loop_adapter.py Adds external-agent dispatch for agentic adapters.
evalscope/api/agent/types.py Splits base/native agent config and adds kwargs.
evalscope/api/agent/trace.py Extends trace metadata and run events.
evalscope/api/agent/loop.py Marks native framework and aggregates usage.
evalscope/api/agent/environment.py Adds per-command env override to environment API.
evalscope/api/agent/__init__.py Exports new agent config types.
evalscope/agent/strategies/react.py Makes ReAct explicitly inherit AgentStrategy.
evalscope/agent/runner.py Adds native agent runner orchestration.
evalscope/agent/external/runners/mock.py Adds mock external runner.
evalscope/agent/external/runners/base.py Adds external runner protocol models.
evalscope/agent/external/runners/_assets/claude_code.Dockerfile Adds Claude Code runner image asset.
evalscope/agent/external/runners/_assets/build_claude_code_image.sh Adds image build helper.
evalscope/agent/external/runners/__init__.py Registers/exposes built-in runners.
evalscope/agent/external/helpers/patch.py Adds tracked-file patch extraction.
evalscope/agent/external/helpers/__init__.py Exports external helpers.
evalscope/agent/external/config.py Adds external agent/bridge config.
evalscope/agent/external/bridge/sse_openai.py Adds OpenAI chat SSE synthesis.
evalscope/agent/external/bridge/sse_anthropic.py Adds Anthropic SSE synthesis.
evalscope/agent/external/bridge/_sse_common.py Adds shared SSE chunking constants.
evalscope/agent/external/bridge/__init__.py Exports bridge server/session.
evalscope/agent/external/__init__.py Exports external-agent APIs.
evalscope/agent/environments/local.py Supports command env overrides.
evalscope/agent/environments/enclave.py Adds aliases, env injection, config merging, and duration handling.
docs/zh/user_guides/sandbox.md Updates agent guide link.
docs/zh/user_guides/agent/native.md Adds Chinese native agent guide.
docs/zh/user_guides/agent/index.md Adds Chinese agent overview.
docs/zh/user_guides/agent/bridge.md Adds Chinese bridge guide.
docs/zh/index.md Updates Chinese docs toctree.
docs/zh/get_started/visualization.md Updates Chinese trace docs link.
docs/zh/get_started/parameters.md Updates Chinese agent parameter links.
docs/en/user_guides/sandbox.md Updates agent guide link.
docs/en/user_guides/agent/native.md Adds English native agent guide.
docs/en/user_guides/agent/index.md Adds English agent overview.
docs/en/user_guides/agent/bridge.md Adds English bridge guide.
docs/en/index.md Updates English docs toctree.
docs/en/get_started/visualization.md Updates English trace docs link.
docs/en/get_started/parameters.md Updates English agent parameter links.

Comment thread evalscope/web/src/pages/EvalTaskPage.tsx
Comment thread evalscope/web/src/pages/PerfTaskPage.tsx
Comment thread evalscope/api/agent/types.py
Comment thread docs/en/user_guides/agent/native.md Outdated
Comment thread docs/zh/user_guides/agent/native.md Outdated
Comment thread evalscope/web/README.md Outdated
@Yunnglin
Yunnglin merged commit 639eb33 into main May 25, 2026
3 checks passed
Abhijnya002 pushed a commit to Abhijnya002/evalscope that referenced this pull request Jun 1, 2026
@Yunnglin
[Feature] Add agent bridge and webui update (modelscope#1364)
00365c0
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
@Yunnglin
[Feature] Add agent bridge and webui update (modelscope#1364)
a808af8 
* feat(external-agent): P0 bridge for third-party CLI agents

Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.

* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
  AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
  gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
  lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)

* rewise p0

* update p0

* update p0

* update registry

* update p1

* update web view

* simplify web view

* openai sse proxy bridge

* update webui refactor

* refactor(web): warm-cream light theme + per-theme score vars

- Light theme reworked from cool-slate translation to warm-cream Console:
  surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
  (#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
  tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
  yellow mid-tones stay legible on cream without forking the brand HSL.
  scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
  switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
  CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).

* openai responses api bridge + codex runner

Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.

* translate_responses.py: full coverage of codex/OpenAI input items —
  message / function_call / function_call_output / reasoning /
  custom_tool_call(_output) / computer_call(_output) plus opaque
  placeholder rendering for built-in tool items (web_search_call,
  mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
  image_generation_call, local_shell_call). item_reference + unknown
  types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
  event sequence (created → in_progress → output_item.added/done × N
  → completed), strictly monotonic sequence_number, 32-char
  function_call_arguments chunking matching inspect_ai. Upstream error
  path emits OpenAI SDK ResponseErrorEvent shape (flat
  type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
  and _prepare_sse_response helpers extracted; anthropic 401 shape
  preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
  new messages_key parameter; _ingest_initial_user_message filters
  type=='message' to avoid mis-ingesting function_call_output as user
  prompts; _extract_responses_tool_results WARNs once when an output
  appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
  -c model_providers.evalscope.* config injection, npm install
  fallback for codex CLI. Setup + run + answer extraction via
  --output-last-message.

Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.

* update codex bridge

* external-agent: review cleanup, trace fixes, agent docs reshuffle

Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
  drop output-last-message kwarg (constant). Remaining knobs: model_name,
  extra_args, extra_config, home_override, auto_install, install_timeout_s,
  node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
  with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
  so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
  kept as the default sentinel for callers that already passed it.

Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
  responses-API turn (top-level instructions + every system/developer item
  + every user item, in document order), not just the first user message.
  Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
  and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
  already-recorded ChatMessageTool entries, replacing the tail-only
  shortcut. codex re-sends every (function_call, function_call_output)
  pair interleaved through input[], not clumped at the tail, so the old
  logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
  results as new events.

Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
  -> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
  empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
  (subsumed by qwen_via_claude_code + walking_skeleton). Rename
  test_responses_instructions_merge.py -> test_responses_input_assembly.py
  to match what it actually covers.

Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
  (overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
  (external CLI bridge: claude-code, codex). Cross-link from sandbox /
  parameters / visualization / READMEs / index.

Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
  run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
  with no in-tree references; their code paths are now fully covered by
  tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
  TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.

Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
  (modelscope/ms-enclave#12)

* update codex bridge
@Yunnglin
Yunnglin deleted the add/agent_bridge branch June 10, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin