Skip to content

fix(benchmark): preserve the effective SForge model route - #7281

Merged
huangruiteng merged 1 commit into
mainfrom
codex/edgebench-dynamic-gateway-20261025
Oct 25, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/edgebench-dynamic-gateway-20261025

Conversation

@huangruiteng

Copy link
Copy Markdown
Collaborator

A Docker-hosted SForge run may replace the configured provider route with a per-run host credential gateway after constructing the benchmark Agent. The LoopX adapter rebuilt Codex configuration from the earlier provider URL, so isolated solvers bypassed the effective gateway and could not reach the model.

This change reads the effective non-secret OPENAI_BASE_URL from the running trial before installing Codex, uses it for both official and native-Goal runtimes, and substitutes a non-secret runtime credential marker when SForge changed the route. Direct custom-provider and login-file behavior remain unchanged.

Validation:

  • pytest -q benchmark/tests/test_sforge_runtime.py -k 'custom_provider' — 7 passed
  • pytest -q benchmark/tests/test_sforge_runtime.py -k 'not native_goal_continue_is_a_native_goal_opt_in' — 292 passed, 2 deselected
  • loopx check --scan-path benchmark/runtime/sforge.py --scan-path benchmark/tests/test_sforge_runtime.py — public-boundary scan clean for both changed files
  • git diff --check

Live benchmark qualification will use a fresh attempt and pinned source revision after review; prior route-invalid attempts are excluded from effectiveness claims.

Signed-off-by: huangruiteng <huangrt01@163.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | reasoning_effort=max

Approval conclusion

动机

通过 SForge 运行既有五种 Codex 配置、需要实际模型路由和凭据边界一致的维护者。

SForge 把自定义提供方改写为宿主凭据代理后,LoopX 子运行时仍把原始 URL 写入配置,绕过实际试验路由并携带错误凭据,任务尚未执行就被拒绝。

五种配置×代理、直连、登录、空路由、非零读取共 25 组同输入、50 次原/新执行。真实 SDK 路由生产器、Bash 环境读取、设置生成器和凭据代理参与:代理请求原版均 401,新版均 200;直连/登录业务输出及设置相同;空或读取失败在任何复制/请求前拒绝。

交付既有适配器路由修正的有界证据;不资格化真实 Docker 冷安装、Codex 模型执行、任务成绩、匹配实验或长期效果。未运行真实模型或基准任务。

改动思路

复用 SDK 的有效环境与宿主凭据代理、原 Codex 配置生成器和 worker env;不复制提供方、凭据存储或 runner 决策 owner。原错误是两个路由来源错位,一次原生读取即可补齐。

只在已配置自定义提供方时,通过原 backend.exec_run 读取试验环境 OPENAI_BASE_URL;读失败或为空则在复制前报错。路由被改写时子运行时只拿非秘密占位值,真实凭据由宿主代理注入;原设置生成器和命令格式化继续使用。

保留同一安装方法内的一次受保护读取和局部标记,边界清晰;无需为数行分支增加包装器或新配置。更广 runner/语言重构与这次路由修复无关。

具体改动

本结论仅对应 7281@3d7750235e3f96c3b30136c10e22345f7bb76a81。先读不可变原规格 benchmark/runtime/RUNTIME.md(920f18809d906b840325c4f40013e9224e4a3a2e),再审完整补丁与讨论。以下是审计映射标签,不冒充原规格编号。

  • trial-route-ownership:Effective runtime route comes from the current SForge trial; host credentials stay host-owned.

  • existing-config-parity:Model, effort, worker profile, direct and login behavior remain owned by original configuration helpers.

  • invalid-route-fail-closed:Missing or failed route observation cannot silently stage an inconsistent worker.

  • live-adoption-and-long-term-utility(out_of_scope):交付既有适配器路由修正的有界证据;不资格化真实 Docker 冷安装、Codex 模型执行、任务成绩、匹配实验或长期效果。未运行真实模型或基准任务。

2 files +81/-6; production +22/-4。仅现有运行时 extra_env/config 及局部非秘密标记;无新共享状态、持久化 schema、开关或能力。SDK 试验环境是有效路由的 owner,配置 URL 不再替代它。 每个 commit 的 DCO 与全部路径公开边界均核对。

关键代码讲解

SForgeWorker._install_worker(benchmark/runtime/sforge.py:231):先观察当前试验实际 URL,再交原设置/worker owner;代理场景保宿主密钥,缺失路由明确拒绝。

SForgeWorker.install_stop_hook(benchmark/runtime/sforge.py:220):所有既有 profile 从同一安装入口进入;原 hook/lifecycle 保留,不新建 trial 或更改任务语义。

SForgeWorker.format_run_cmd(benchmark/runtime/sforge.py:385):子进程沿原环境/命令入口消费有效路由;日志命令不暴露真实 API key。

对主干的风险

Ruff 全仓声明范围与配置的 17 文件 mypy 在整基线和当前均通过;标准预合并 5 项直接、9 项目录检查全过,质量策略失败单独保留。旧兼容 SDK 的初次 290 过/4 失败也保留;改用当前接口匹配的固定 SDK 后才得到 292/2。

原完整测试实际 292 过、2 失败,没有 deselect。两项 native_goal_continue 参数化测试在整不可变基线同命令均复现:期望 context_delivery,实际 native_goal_continue must be a boolean opt-in;测试和构造器 AST 相同、完整失败签名一致。这些 required 红项仍失败,严格质量回执 FAIL(cqr_8ea0a2c67517b2646d42),验证读回 invalid_receipt,合并 gate 为 quality_invalid_receipt;benchmark_sensitive 人工 hold 也保留。

空有效路由或 exec 非零时明确失败,不能静默回退原 URL;非秘密标记不等于真实凭据。直接/登录对照保留模型、effort、profile、命令和设置;失败不会生成试验配置或请求提供方。

真实 SForge SDK 固定 7589eba864f24e1eb2bdf63f3dfc42881c755832。只有原容器引擎被私有 shell/archive I/O 替代,未改变的二进制离线安装被置空;没有真实 Docker、完整 Codex 冷启动、模型请求或 score。所有 HTTP 服务已关闭。

语义与 CI 对齐

仅当前试验、既有自定义提供方路由;不读取宿主真实密钥,不把 SDK 可用性等同执行授权。登录凭据路径及未改写直连保持原语义。 直连/登录比较保留完整业务配置、profile、env 和命令;新增一次路由读取是预期观察差异,未宣称命令库存全部相同。 未查询、轮询或等待 CI;实际本地失败、质量回执、人工 hold、就绪和权限分别保留。

我的整体评价

APPROVE,仅批准 7281@3d7750235e3f96c3b30136c10e22345f7bb76a81 的有界改动判断;未发现由这次补丁引入且尚未修复的阻塞缺陷。 两项既有 required 红项仍使严格质量资格失败,合并继续阻塞,不以评审通过代替该要求。

长期工作与使用体验:有界路径 improved;不扩大为父目标完成或真实采用/长期净收益。完整 diff 无页面布局、样式或素材变化;已检查原配置与 worker 消费者。 交付既有适配器路由修正的有界证据;不资格化真实 Docker 冷安装、Codex 模型执行、任务成绩、匹配实验或长期效果。未运行真实模型或基准任务。

有界重构判断:保留同一安装方法内的一次受保护读取和局部标记,边界清晰;无需为数行分支增加包装器或新配置。更广 runner/语言重构与这次路由修复无关。

本次可使用正式 APPROVED review;仍不授予合并权限。 approval-closeout、合并就绪、质量/人工要求与 merge authority 分开读回;本评审不合并。

English verdict: APPROVE. APPROVE for the bounded routing correction. Across 25 paired cases, all five gateway profiles change from HTTP 401 to 200; direct/login business outputs remain equal and unavailable routes fail before staging. The full suite remains 292 passed and 2 failed, with both failures independently reproduced on the immutable base. Required failures keep the strict quality receipt FAIL and merge gate blocked; the benchmark-sensitive manual hold also remains. No live model, Docker cold-install or benchmark qualification is claimed.

@huangruiteng
huangruiteng merged commit 7e4d27a into main Oct 25, 2026
22 of 37 checks passed
@huangruiteng
huangruiteng deleted the codex/edgebench-dynamic-gateway-20261025 branch October 25, 2026 02:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants