Repository navigation
feat(research): supply native result context to trajectory analysis - #7272
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
There was a problem hiding this comment.
Reviewer: model_agent; model=gpt-6.1-sol; provider=OpenAI; reasoning_effort=max; declaration_source=runtime_reported; observation_id=23a04ec78b16bbedc99165a64bbd64958da29757ab4b99198bfb1ce25350843f
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机
研究者让现有分析器解释已捕获实验片段时,分析器可能缺少页面上已有的原生结果上下文。 原来仅登记原生结果供页面显示;现在维护者在已核验输入中显式绑定该来源后,分析器收到同窗原生结果,旧输入仍产生相同提示词和分析键。 真实 HTTP/store/分析子进程路径已证明上下文送达、重复请求复用、源变化阻止新执行;36组省略绑定对照完整字节一致。 不扩大评估样本或引用范围,不修改原生评分、实验权限与执行预算;上下文送达不等于研究结论正确。 真实模型对该上下文的理解、独立校准、完整成本、因果收益及已安装用户采用仍未证明。
改动思路
原生上下文的检查与投影继续属于已有 TypeScript 分析 owner;Python 仅保留原有文件、进程和HTTP适配,不引入第二评价器或新的执行权限。 本PR只给显式绑定的新分析输入增加同源上下文、身份和重评估规则;旧包与只读展示保持,现有明确分析动作和失败恢复直接复用。 源由既有可信注册路径核验,分析仍是已有明确动作。上下文专用原始JSON不重复进入编号证据;若同源本来属于episode或显示解释则保留原证据。新增说明只有绑定后进入同一observer,未绑定包保持原字节。
具体改动
完整审查6路径 +190/-7,准确提交 40faee717b8d3cdb00f694a0527b94cfbeb59853;实际merge-base bf21a2cf1f5783c34acf4c2796c6cb0205192731,目标main快照为 984691c6ac54686d6747f4325564e68d81397f85。规格 docs/reference/trajectory-assessment.md 固定在修改前 bf21a2cf1f5783c34acf4c2796c6cb0205192731,不使用本PR新增段落自证。映射标签是本次审查标签。TA-Source implemented:Analysis input is source-bound, with scope, full artifact hash and literal captured evidence; source changes prevent new execution while historical readback remains available. 验证:57 real HTTP/store/process tests; explicit-context changed-source409。 TA-Scope implemented:Every assessment and synthesis citation remains within its declared episode and supplied source range; missing evidence stays unknown. 验证:63 focused TS tests including out-of-episode native-capture citations。 TA-Identity implemented:Substantive input/config changes require reassessment; unchanged registered analysis may reuse only its content-bound identity. 验证:same-source-collection rebind test;36 exact base/head off pairs。 TA-Authority implemented:Registration/readback does not launch a model; analysis requires an explicit action and remains an uncalibrated sidecar, never the native evaluator or experiment authority. 验证:57 real API/store/process tests: no spawn on registration/GET/language read; explicit POST only。
关键代码讲解
-
nativeContext:核对已登记原生捕获。新增绑定source id到已有common projector;关键边界是 Full verified source and same Goal/case/run/revision/window; conflict rejected。调用 projectTrajectoryNativeOutcomes 后交给 parseTrajectoryInput and prepareTrajectoryAssessment,失败/恢复由 Malformed/missing/foreign capture rejected by EffectRuntimeRequestError 负责。
-
prepareTrajectoryAssessment:把上下文交给原分析器。新绑定加入prompt和identity;只读native projection仍不改旧identity;关键边界是 Absent binding preserves bytes; bound/display captures must agree; context does not widen episode source_ids。调用 nativeContext, unchanged prompt/schema and citation owners 后交给 existing child process provider prompt,失败/恢复由 Invalid context rejected; source changes blocked by store; existing historical readback 负责。
-
evidenceContent:控制后续版本能否复用。把绑定id加入已有canonical evidence identity;关键边界是 Rebind requires full reassessment even if source collection unchanged。调用 compareTrajectoryAssessments 后交给 existing continuation/register base-pack route,失败/恢复由 No cached judgment reuse after substantive input change 负责。
类型只增加可选来源id;文档说明显式绑定、同捕获冲突拒绝、只读兼容、显式重评估与移除绑定的回滚。实际新API用例记录子进程收到的stdin,原生方向、最佳、最后、未知保留和未知收益均保留。63项TS与57项Python通过;36组省略绑定的完整prepared对象在准确基线/当前逐字节相同,包括语言、synthesis、显示native和历史来源分支。新投递断言在实际基线因缺少上下文失败,在当前通过,能识别原缺陷。
对主干的风险
最强反例是上下文被误当新增评估样本,或者只改绑定却复用旧结论。当前保留每个episode自己的source_ids,metric与synthesis引用上下文专用来源都会被原checker拒绝;更换绑定即使来源集合不变也会触发全量重评估。缺失来源、first_line非1、异scope、坏JSON和显示/分析捕获冲突均拒绝。实际HTTP测试还证明注册/GET/语言读取不启动provider,只有既有明确POST启动一次;重复复用,来源改变409且历史读回保留。没有新增实验、评分、权限或预算所有者。
全量Ruff仍有5处未改 tests/test_native_steward_monitor.py 的E701。准确基线 bf21a2cf1f5783c34acf4c2796c6cb0205192731 与当前 40faee717b8d3cdb00f694a0527b94cfbeb59853 运行同一完整命令,2520字节诊断完全一致,SHA-256 619b9f189e48db446b2e03aef2591415211c07a27c88f1b6da11586814b55d2e;该文件与分析调用链均未改,当前变化另由上述真实路径与负例覆盖。因此判为 pre_existing_unrelated;原检查仍failed,严格质量回执 cqr_c9b47d5c2762db2f6ea1 为 FAIL。没有调低门槛或把失败改成pass。
语义与 CI 对齐
可选字段扩展已有typed输入,不是新的状态枚举或执行权限。新提示词逐条保留样本/引用范围、未知、保留与最终区别及不授予实验权限;真实子进程送达不能证明模型理解或引用蕴含正确。TS全量typecheck、mypy17个配置源文件、chat构建和quick premerge的5项直接/2项catalog/1项公开边界检查通过。按当前策略没有查询或等待远端CI。
我的整体评价
APPROVE,这是可用的 justified_increment,余下研究质量资格没有被宣称完成。long_horizon 为 preserved,user_experience 为 improved。复用已有注册、明确分析、失败恢复与结果读回旅程;已知来源由 curator 绑定,无新增设置、手工同步或用户确认轮次。 真实模型对该上下文的理解、独立校准、完整成本、因果收益及已安装用户采用仍未证明。 全部6个改动路径只改变类型、分析提示词/身份、文档和既有API测试;原有renderer与交互控制不变,实际现有API旅程已验证。 此次没有渲染组件变化;真实模型解释质量、已安装 App 的实际采用、完整费用与因果效果未验证。 已有真实API、注册与明确分析旅程足够承载这次输入改进,无需新增设置页或把源id交给普通用户手填。
省略绑定的36组基线/当前比较不做归一化,完整提示词、键、schema及投影一致;显式绑定才增加语义,严格质量仍因原有Ruff保持FAIL,与本次变更审查归因分开。现有owner和一个可选选择器足够,没有需在当前PR扩大或拆分的框架重构。公开APPROVE不等于质量全绿、GitHub正式自我批准、合并准备或维护者合并授权。未执行合并、共享安装、实验任务或真实模型调用;新提交、来源/调用者变化需重新审查。
English verdict: APPROVE — 40faee717b8d3cdb00f694a0527b94cfbeb59853. Explicit verified native context reaches the existing analysis process without widening episode citations; omitted bindings preserve complete prepared bytes in 36 base/head pairs. 63 TS and 57 real API/store/process tests pass. The unchanged five Ruff E701 diagnostics match the immutable base exactly; strict change-quality remains FAIL. Model-quality and installed adoption are unverified; no merge performed.
Research can display native measurements while its observer only sees sampled episode excerpts. A curator can now bind a complete registered native capture with
native_outcomes_source_id; the existing typed native owner supplies metric direction, coverage and the distinction between best, final and retained results to the same analysis request. Unknown retention remains unknown. This completes the S5/S11 reader’s measured-results-to-interpretation input path without adding a second evaluator.The binding and its instructions participate in analysis identity and continuation reuse. Changing the binding requires reassessment even when the source collection is unchanged. Episode and synthesis citation scopes remain unchanged. Existing display-only registrations retain their prompts, keys and saved results; enabling this context requires a new explicitly bound input and an explicit analysis action.
Validation on
40faee717b8d3cdb00f694a0527b94cfbeb59853againstbf21a2cf1f5783c34acf4c2796c6cb0205192731:The same source filter now handles context-only captures and display-only snapshots; episode/display citation sources retain their numbered text. No new configuration owner, renderer or execution surface was added. Model entailment, researcher comprehension, complete cost and installed adoption remain separate qualification gaps. This proposal is for review; it is not merged or installed.