Skip to content

fix(worker): preserve original replan through bounded Host retries - #7241

Merged
huangruiteng merged 3 commits into
mainfrom
codex/replan-host-retry-main
Oct 24, 2026
Merged

huangruiteng merged 3 commits into
mainfrom
codex/replan-host-retry-main

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and change / 问题与改动

A transient Host failure left an unknown original replan even when its bounded transport policy permitted retry. Retain the dispatch obligation and reuse exactly one matching original semantic attempt and session after fresh Core/Host admission.

Also preserve the opposite recovery direction: the original refresh and spend may already be complete when the final Host response fails. Read the exact original canonical settlement, including after a saved recovery marker survives restart, and permit subsequent admitted work only when one exact dispatch-bound original attempt is currently verified settled. Missing, conflicting or unreadable proof, ambiguous or legacy bindings, stopped workers and exhausted policies remain held. Transport success and local journal markers never acknowledge semantic completion.

Host 暂时失败后仅复用唯一匹配的原 obligation、attempt 与会话。原 refresh/spend 已完成但最后回包丢失时,必须重新核验原 canonical settlement 后才能继续后续工作;重启后的本地恢复标记不能替代证明。保留停止、退避、次数上限、未知与歧义保护,不重复原结算。

Scope and provenance / 范围与来源

Independent main port of #7133, followed by the original-settlement recovery correction requested on #7241. Reuses the existing Host policy and TypeScript settlement owner; no checkpoint-port dependency, new Goal state, model/account change, scoring change or benchmark launch.

Validation / 验证

Reviewed head: b36b4f3a2b99b252d3d48d920b68fd1b6f209ca4.

  • Worker retry, real File/SQLite CLI and existing worker suites: 145 passed.
  • Exact committed-head recovery subset: 10 passed, including crash after durable recovery, missing/unreadable proof and wrong original identity.
  • Before the correction, both real File/SQLite recovery cases failed at the original head; the same cases pass on the fix.
  • Immutable main worker comparison: 122 existing cases passed. 22 subprocess cases additionally rerun with the worker script itself pinned to the baseline; initial module-only replacement was insufficient for those child processes.
  • ruff, Python compilation, diff hygiene, RFC-pair check and the four-file public-boundary scan passed.
  • Risk-based premerge: 5 direct + 17 selected checks passed; exact-scope change-quality receipt verified pass. Initial premerge failed because the required TypeScript development package was absent; after npm ci --ignore-scripts, semantic validation and the full premerge gate passed. The failure is retained as setup evidence, not relabeled.

The configured review policy uses local validation without CI waiting. Earlier remote failures on the old head are not rewritten. Actual provider capacity, installed-package behavior and long-horizon benefit are untested. Host responses and backoff clocks are synthetic; Core admission, File/SQLite authority, original refresh/spend and settlement identity readback are real. This proposal and its review do not themselves execute or authorize a merge.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | reasoning_effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机

由外部 shell worker 持续执行已获准工作的用户。

Host 暂时容量失败后,同一语义重规划被记录为 unknown;旧 worker 的未知结果保护会阻止即使允许的同配置有限重试。

当前改动使同一未完成重规划能续用原会话重试;但如果该次工作其实已完成 refresh 与 spend,Host 的最后响应才失败,下一次 Core 已准入后续工作时,worker 仍把它挡住。

实际 File 和 SQLite 同源配对均证明两个方向:同一未完成 obligation 基线停止、当前重试成功;原结算已确认且后续工作已准入时,基线继续 exit 0,当前 exit 2/original_replan_not_retryable,未调用第二次 Host。

只审可选 --wake-host-failure 适配器、同配置有界重试和既有语义结果恢复;Host 退出码不是语义 ACK,transport journal 不是 Goal/Todo/配额 authority。必须保留原身份、会话、有限次数、backoff 和 Core stop。

改动思路

transport retry 应复用历史 dispatch 关联,语义完成必须读取原 canonical settlement,两种事实应保持各自 owner。派生一条 dispatch 关联有依据,但复核原结算后仍用“没有 unknown 可重试”阻断 fresh admission,把完成误当未完成,越过了既有恢复 owner。

新增 helper 只复用同 scope、同 obligation、唯一 unknown 的原语义 attempt;dispatch 前把 obligation 关联写入 host_attempt,并沿用原 session/attempt budget。新的 guard 在没有可复用 attempt 时无条件拒绝,却没有区分原结算已经被既有恢复 owner 验证完成的情况。

保留严格的未完成/含糊/跨 scope/缺 session/非 retryable 拒绝;只在原 settlement 身份被既有 owner 精确验证为 settled 后,允许当前 Core 准入的工作继续,并按原策略处理 transport 状态。不以普通 changed admission 或 Host success 代替该证明。

具体改动

[P1] 当原工作已经通过正常 CLI 完成 refresh/spend,但 Host 最后的响应返回可重试容量失败时,下一次 Core 可以准入后续 Todo。紧前面的 _read_resolved_replan_attempts 此时已经精确确认原身份为 settled,并把 attempt 写为 returned/original_settlement_verified;_host_retry_replan 因为它不再是 unknown(且当前通常是 normal_run)返回 None,这个新增 guard 就仍然 exit 2。独立真实 File/SQLite 同种子验证均为基线第二次 Host 调用/exit 0,当前仅首次调用/exit 2,且磁盘保留上述恢复证明。请区分“未知原 obligation 不能重试”和“原结算已验证完成”:仅后者按既有恢复策略继续当前 Core 准入的工作;保持其他保守 hold、原会话、预算及身份不变,并增加两种 provider 的该恢复回归。

不可变已接受规格:docs/reference/shell-worker.md,版本 7c48375df5aa4824c8d6fba0baef1f96de6cc87e。先读原规格再审补丁;下列为本审计映射标签,非伪造原编号。

  • bounded-original-host-retry(implemented):Only retryable failures with the retained original session and bounded same-configuration policy may retry after fresh Core admission.

  • original-semantic-recovery(not_met):Original canonical settlement proof resolves a lost response and permits subsequent admitted useful work.

  • stop-off-unknown-isolation(implemented):Stop dominates; unresolved/legacy/ambiguous states retain their hold; the optional adapter stays default off.

4 files +159/-10; production +36/-6。完整差异、相关既有 owner、全量讨论、DCO 与公开边界已核验。新增关联由本次实际 dispatch 从现有 replan attempt 派生并在调用前落盘;legacy/unbound/started/ambiguous 状态继续保守 hold。新增字段不授予语义结算权,但其恢复/退休必须服从原 settlement proof,不能永远残留阻断。

关键代码讲解

_host_retry_replan(scripts/external_scheduler_worker.py:383):同 scope 和 obligation 下选择唯一 unknown 原 attempt;不会建立新语义结算身份。

_read_resolved_replan_attempts(scripts/external_scheduler_worker.py:405):使用原 effect/Turn/obligation 读取真实 canonical settlement,精确已结算时落盘 returned/original_settlement_verified;这是现成的恢复证明。

run_worker retry admission(scripts/external_scheduler_worker.py:859):新 guard 只接受可复用 unknown;已恢复 returned 因而同样被拒绝,尽管 Core 已准入后续工作。

run_worker dispatch intent(scripts/external_scheduler_worker.py:899):Host 调用前落盘 replan_obligation_id,沿用原 session 与 attempt budget;这一历史关联须在已验证恢复后有正确继续路径。

对主干的风险

专项135通过;标准预合并5项直接/编译与17项选中检查通过。额外同源 File/SQLite 的同 obligation 重试均为基线1次/当前2次,保留唯一语义 attempt 与原身份;adapter-off 两种 provider 完整 worker journal 与拒绝行为配对一致。

第一次真实 Core admission 后用正常 CLI 创建 successor 并完成原 refresh/spend,再返回合成 retryable provider_capacity。30秒 backoff 后真实 Core 允许后续 Todo,_read_resolved_replan_attempts 已保存 returned/original_settlement_verified。基线第二次调用 Host,当前却被新 guard 拒绝;File、SQLite 各1对均如此。

这是新增 guard 引起的实际恢复回归,135项测试与标准 canary 通过不能消除。Host response 是合成类型化 envelope;准入、canonical File/SQLite、原 refresh/spend 和 settlement readback 都是真实 CLI/owner。没有 live provider 容量测试、安装版或长期净收益资格;未查询 CI。

基线与当前只替换 worker 源码;两种 provider 分别还原相同物理种子。合成边界仅为类型化 Host 返回包和 worker backoff 时钟;真实 CLI、Core、canonical authority、原 refresh/spend 与读取身份均未替代或归一化。

语义与 CI 对齐

host_attempt.replan_obligation_id 是持久化的历史 dispatch 关联,源自当前 canonical admission/原语义 attempt;不是另一个 obligation owner,也不是人工同步配置。新的 original_replan_not_retryable 是本地诊断,不是共享 authority enum。 原结算恢复是现有义务;新 guard 的反例违反该义务,应在原 owner 边界修复并重跑实际恢复、same-obligation/off/negative cases。 CI 按契约未查询、轮询或等待。

我的整体评价

REQUEST_CHANGES,仅对应 7241@4cb0fedeb6ee7013f99d6b2b1f061e51a2c104da。当前未能满足完整恢复目标;请修复已验证原结算后的继续路径,再审新的完整 head。

长期工作:regression;使用体验:regression。实际 File 和 SQLite 同源配对均证明两个方向:同一未完成 obligation 基线停止、当前重试成功;原结算已确认且后续工作已准入时,基线继续 exit 0,当前 exit 2/original_replan_not_retryable,未调用第二次 Host。

已检查实际 CLI/Host caller、完整输出、失败与恢复路径;没有页面、布局或视觉素材改变,故不需截图。 未验证范围:这是新增 guard 引起的实际恢复回归,135项测试与标准 canary 通过不能消除。Host response 是合成类型化 envelope;准入、canonical File/SQLite、原 refresh/spend 和 settlement readback 都是真实 CLI/owner。没有 live provider 容量测试、安装版或长期净收益资格;未查询 CI。

有界重构判断:保留严格的未完成/含糊/跨 scope/缺 session/非 retryable 拒绝;只在原 settlement 身份被既有 owner 精确验证为 settled 后,允许当前 Core 准入的工作继续,并按原策略处理 transport 状态。不以普通 changed admission 或 Host success 代替该证明。 当前不增加新框架、领域 owner 或仪式化后续任务。只审可选 --wake-host-failure 适配器、同配置有界重试和既有语义结果恢复;Host 退出码不是语义 ACK,transport journal 不是 Goal/Todo/配额 authority。必须保留原身份、会话、有限次数、backoff 和 Core stop。

这是 author-owned COMMENTED 结论,不是 GitHub 正式 self-approval/rejection;approval-closeout、readiness、人工 hold 和合并权限另行读回。本评审不执行合并。

English verdict: REQUEST_CHANGES. REQUEST_CHANGES. The bound unknown-attempt retry works, but the new guard also blocks after canonical proof has resolved the original settlement and Core has admitted subsequent work. Both real File and SQLite base/head comparisons reproduce this regression. Reuse the existing exact original-settlement recovery proof without weakening unknown, scope, session or retry-budget holds.

Comment thread scripts/external_scheduler_worker.py
…61025

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | GPT-6 | OpenAI | self_reported

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机

使用外部 shell worker 持续执行已获准工作的用户。
Host 暂时容量失败会留下结果未知的原重规划,既有保护使同配置有界重试也无法继续。
同一未完成重规划现在可在原会话中继续;原工作已经结算但最后响应丢失时,也能恢复后续已获准工作。
此 PR 只改变可选 Host 失败适配器的恢复,不改变任务选择、语义结算、配额、模型或账户权限。

改动思路

传输重试与语义完成分别读取各自 owner。Host policy 负责原会话、次数与退避;Core 的原始结算读取负责证明原工作完成。worker 仅在 dispatch 前保存原 obligation 的关联,不用当前新准入推断旧工作结果。

同一未知 obligation 可复用唯一原 attempt。对于已经完成的原工作,恢复 reader 返回本次真实核验通过的对象,guard 只接受其中与原 dispatch 和当前作用域精确匹配的唯一对象。重启后的本地 recovered 标记也要重新读取 canonical proof。继续后续工作时不重放原 refresh/spend。

保留设计的理由是沿用现有结算 reader 和 Host policy,只补上两者的关联,不建立第二个结算 owner。当前 PR 边界是可选 shell Host 适配器的有界恢复。直接删除 guard 会放行不明旧效果,只保持旧逻辑则会让已完成工作永久阻塞;这两种更小表面修改均不满足恢复合同。这里使用文字说明分支,不提供未经渲染验证的示意图。

具体改动

评审完整 head:b36b4f3a2b99b252d3d48d920b68fd1b6f209ca4。相对上一轮 4cb0fedeb6ee7013f99d6b2b1f061e51a2c104da,修复的是已验证原结算后的错误 hold;当前整份 PR 仍包括同 obligation 原会话重试及文档、回归用例。

不可变规格:docs/reference/shell-worker.md,规格版本 7c48375df5aa4824c8d6fba0baef1f96de6cc87e。以下沿用评审映射标签,不声称是原文编号:

  • bounded-original-host-retry:implemented。显式 retryable、保留原 session、同配置与有限次数/退避约束均保留;相同 unknown obligation 复用一个语义 attempt。
  • original-semantic-recovery:implemented。只有本次原 canonical settlement 的精确身份与 settled 结果证明,才允许后续已获准工作继续;本地标记或新准入不能替代该证明。
  • stop-off-unknown-isolation:implemented。停止优先;legacy、歧义、错作用域、无 session、硬失败与次数耗尽继续保持 hold;普通未启用路径保留原行为。

关键代码讲解

  • _host_retry_replan:只选择同 scope、同 obligation 的唯一 unknown,续用原 attempt,不生成新的语义尝试。
  • _read_resolved_replan_attempts:调用既有 TypeScript settlement owner,验证完整原身份与 settled;返回本次核验的对象。与原 Host failure 关联的历史恢复标记也重新核验。
  • _run_worker_locked:先通过既有 Host retry/backoff,再读取原结算;只有唯一且精确的原对象存在于本次核验结果中,才能解除新增 guard 对后续工作的误阻断,其他 pending attempt 仍阻止 dispatch。
  • test_settled_replan_host_failure_allows_only_verified_next_work:真实 File/SQLite CLI 创建 successor、refresh、spend、下一 Turn 准入,再丢失外层响应。覆盖恢复写盘后重启,以及证明缺失、不可读、错误原身份;核验原 spend 恰好一次。

对主干的风险

专项命令 uv run --extra test python -m pytest -q tests/test_external_scheduler_replan_retry.py tests/control_plane/test_external_worker_replan_cli.py tests/test_external_scheduler_worker.py:145 passed。精确提交上单独重跑新增恢复用例:10 passed。修复前,同一新用例在 File 和 SQLite 上均因错误 hold 失败;修复后均通过,未删除失败断言。

以不可变主干 b4606549be7c4ef146ba65467caefe360d122940 的 worker 替换执行对象,其余相同测试/CLI/TS owner 保持一致:122 项既有用例通过。涉及独立子进程脚本入口的 22 项另外将 script 路径也钉到基线后复跑通过;最初替换仅覆盖模块对象,不能把其子进程调用声称为基线,已补齐该缺口。比较覆盖普通 opaque 命令、停止、硬失败、退避、session、崩溃恢复、单消费者锁及未知语义效果。输入参数、完整拒绝提示与持久状态断言由原测试保留;只归一化测试时长与日志时间,不归一化语义结果。

Ruff、Python 编译、diff/RFC 配对检查、四文件公开边界扫描及语义词汇检查通过。预合并首次执行的 TypeScript 依赖缺失失败保留;安装声明的开发依赖后语义检查通过,完整预合并重跑的 5 项直接检查与全部 17 项选中检查均通过;精确范围 change-quality 收据已验证通过。四文件公开扫描使用隔离合成注册表,只验证本 PR 的公开边界,不代表运行环境的总体资格。

语义与 CI 对齐

原 obligation 关联是 dispatch 派生的本地传输证据;既有 recovery 标记是诊断,二者都不能替代原 canonical settlement。没有新增共享状态词汇或新的语义结算规则。read_heartbeat_settlement 仍桥接现有 TypeScript owner;Python 改动属于现有 Host/process IO 边界,不要求额外语言迁移。文档明确有界重试、原证明与暂停后果,未把强制 guard 描述成建议。

当前 Goal 配置为不等待 CI;按本地验证评审,不继续获取或轮询远端 CI。早先旧 head 的远端失败记录不改写。真实 provider 容量、安装版行为、付费模型结果及长期净收益未测;合成边界只包括 Host 返回包、worker 时钟和指定 proof 故障,准入、File/SQLite、原 refresh/spend 与 settlement identity 读取均走真实 CLI/owner。

我的整体评价

APPROVE,仅对应 b36b4f3a2b99b252d3d48d920b68fd1b6f209ca4,无阻塞代码发现。长期工作:improved;使用体验:improved。同一未完成重规划可继续原会话,原结算完成后后续工作不再被误挡;不增加用户找身份、重输信息或手工解除暂停的步骤。未知原结果仍需原恢复,不以继续运行取代安全证明。

只改变 shell worker 的失败/恢复分支,没有页面、布局或视觉素材改变,故不需截图。视觉未验证范围:无新增渲染表面。保留的唯一 dispatch 关联有真实调用方与崩溃恢复理由;reader 返回现成 proof 对象的有界整理已应用,无额外框架或状态 owner。评审不等于合并资格、安装或长期收益;精确 head 的 change-quality、预合并与 merge-readiness 仍分别核验。

English verdict: APPROVE. At b36b4f3a2b99b252d3d48d920b68fd1b6f209ca4, bounded retries reuse the original unknown replan, while fresh canonical proof of its exact settlement permits subsequent admitted work. File/SQLite recovery, restart and proof-failure cases pass; ordinary-path and retry/stop guards remain covered. This is an author-owned COMMENTED conclusion, not formal self-approval or merge authority.

@huangruiteng
huangruiteng merged commit 34d9d40 into main Oct 24, 2026
23 of 37 checks passed
@huangruiteng
huangruiteng deleted the codex/replan-host-retry-main branch October 24, 2026 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants