Scope research follow-up designs to the next decision

docs/reference/trajectory-assessment.md

@@ -76,6 +76,47 @@ divergence. A justified repeat with contemporary controls can still be useful.
 These are model reasoning instructions; the validator does not certify that the
 answer followed them or that a proposed experiment is scientifically sufficient.
 
+Optional revision 3 (`config.research_synthesis_revision: 3`, with synthesis
+enabled) adds case-selection reasoning in the existing `why_priority` and
+`controls` fields. When proposing concrete cases, explain their observed
+selection basis, changed baselines, useful supporting or contrary cases, and
+missing evidence. Separate a retrospective diagnostic pilot from a predeclared
+sample and its broader generalization. A pilot need not test every case, and
+matching labels do not establish independent or comparable observations.
+Register a successor pack and explicitly generate its report through the same
+workbench; browsing does not invoke a model. Disable this guidance by choosing
+a revision 2 or legacy pack. Default producer requests remain revision 2, and
+frozen v1/v2 prompts, keys, results, failures and usage are unchanged. Revision 3
+adds no result fields, new model call per finding, solver behavior or scientific
+qualification gate. Its case-selection rationale remains uncalibrated advice.
+
+可选第3版沿既有优先级理由／控制条件说明案例选择依据、基线变化、反例和缺失证据，
+区分事后诊断试点与预先定义样本，不把少量试点外推为普遍收益。接续pack需显式生成，
+浏览不调用模型；选择第2版或旧pack即可停用。默认仍为第2版，旧报告、失败和用量不改；
+新增指引不等于模型已遵从或科学有效性获得认证。
+
+Optional revision 4 (`config.research_synthesis_revision: 4`) retains revision 3
+and asks which decision the next step should change before choosing its design.
+Candidate selection, component attribution and interaction testing answer
+different questions. Reuse completed controls and negative evidence; prefer a
+staged comparison when it can resolve the immediate uncertainty, and state what
+observation would justify a broader design. A factorial or replication is useful
+when the decision needs it, rather than an automatic response to multiple changes.
+Known case/arm/repetition limits and preparation, validation and analysis costs
+belong in the existing priority/intervention fields. Configured allowance is not
+actual elapsed cost; unknown costs remain unknown. Existing evidence can justify
+no new experiment. These are observer instructions, not an enforced minimization
+rule, a scientific qualification or execution authority. Register and explicitly
+analyze a successor pack; select an older pack to disable the new guidance.
+Default requests remain revision 2. Frozen v1/v2/v3 requests, jobs, results,
+failures and usage retain their identities and are never automatically upgraded.
+
+可选第4版先明确下一步要改变的决策，区分方案选择、组件归因与交互检验；复用已完成
+对照和负结果，能分阶段判别时说明先做什么、出现什么证据才扩大设计。更全面的设计
+或复验须说明决策需要；配置预算不当作实耗，准备／验证／分析的未知成本保留未知。
+已有证据足够时不强加新实验。这是观察者指引，不是强制最小化、科学认证或执行授权。
+须注册接续pack并显式生成；选回旧pack即可停用。默认第2版和旧版身份、失败、用量不改。
+
 New default producer requests persist revision 2. Previously stored synthesis
 configurations that omit the revision retain the exact v1 prompt, key and
 language caches; reading or retrying them does not upgrade the method. Explicit

loopx/control_plane/capabilities/trajectory_assessment.ts

@@ -204,6 +204,12 @@ For a resource-limit explanation, distinguish a configured ceiling from attempte
 Use the existing controls, prediction and disconfirming_result fields to make the proposed test interpretable. Name only the operational checks needed to tell whether its intervention engaged and its comparison held, such as phase entry, actual exit reason and completed/effective work. Make predictions conditional on those checks; a failed manipulation is not a refutation of the mechanism.
 Equal seeds or wall-clock limits alone do not establish matched random draws, proposal coverage or completed work. Where relevant, distinguish a fixed-work comparison from an end-to-end wall-clock comparison, and say how actual coupling will be checked. Identify intended post-intervention state/path divergence separately from a confound; do not demand identical later trajectories when changing those trajectories is the mechanism. Sampled proposals, accepted intermediate moves and direct final improvements are different observations; no direct update does not exclude an indirect path effect.
 Keep findings and proposals concise and specific to the cited evidence. These are reasoning instructions, not extra measurements, automatic scientific qualification or permission to execute a test.`;
+const SYNTHESIS_CASE_SELECTION_INSTRUCTION = `${SYNTHESIS_DECISION_INSTRUCTION}
+When proposing concrete cases or subsets, use why_priority and controls to explain their selection from the supplied observations and which competing explanation they can distinguish. Separate observed case differences from future predictions; check whether an apparent gain instead reflects a changed baseline. Where relevant, consider supporting, unchanged and contrary cases, or state which evidence is missing. Distinguish exploratory selection after seeing results from a predeclared sample. A purposive diagnostic pilot can be useful without testing every case, but does not establish prevalence, generalization or a fair ranking. Do not invent representative cases, exhaustive coverage or independence from matching labels.`;
+const SYNTHESIS_DECISION_SCOPE_INSTRUCTION = `${SYNTHESIS_CASE_SELECTION_INSTRUCTION}
+Choose the next decision before its experimental design. Distinguish choosing between candidates from attributing a combined result to components or interactions; a comparison can answer one without answering the others. Explain which decision the proposal can change and what remains outside its scope. Do not expand a candidate comparison into a factorial merely because multiple components changed; justify broader attribution when it is needed for the decision.
+Reuse applicable completed controls and useful negative results before adding work. Prefer a staged comparison that resolves the immediate uncertainty, with an explicit observation that would justify extending it; a larger design or replication remains appropriate when a narrower one cannot distinguish the relevant explanations. If reviewing existing evidence is sufficient for the next decision, say so and do not invent a new experiment.
+In why_priority and intervention, weigh the information needed for that decision against the proposed work. When supported, account for cases, arms, repetitions and per-run limits, plus preparation, validation and analysis; keep configured execution allowance separate from actual elapsed cost and mark missing costs unknown. Do not claim a design is cheapest or optimal without a supported comparison. An informative tie or negative result can settle the immediate decision without assigning the earlier combined gain to individual components.`;
 const groundedSchema = {
   episode_ids: {type: 'array', minItems: 1, items: {type: 'string'}},
   citations: {type: 'array', minItems: 1, items: citationSchema},
@@ -230,11 +236,14 @@ export function prepareTrajectoryAssessment(params: Record<string, unknown>) {
   }
   const synthesis = config.research_synthesis === true;
   const synthesisRevision = config.research_synthesis_revision;
-  if (synthesisRevision !== undefined && (synthesisRevision !== 2 || !synthesis)) {
-    throw new EffectRuntimeRequestError('research_synthesis_revision requires enabled synthesis and supported revision 2');
+  if (synthesisRevision !== undefined && ((synthesisRevision !== 2 && synthesisRevision !== 3 && synthesisRevision !== 4) || !synthesis)) {
+    throw new EffectRuntimeRequestError('research_synthesis_revision requires enabled synthesis and supported revision 2, 3 or 4');
   }
-  const synthesisInstruction = synthesisRevision === 2 ? SYNTHESIS_DECISION_INSTRUCTION : SYNTHESIS_INSTRUCTION;
-  const method = synthesis ? synthesisRevision === 2 ? 'research-episodes-synthesis-v2' : SYNTHESIS_METHOD : TRAJECTORY_METHOD;
+  const synthesisInstruction = synthesisRevision === 4 ? SYNTHESIS_DECISION_SCOPE_INSTRUCTION
+    : synthesisRevision === 3 ? SYNTHESIS_CASE_SELECTION_INSTRUCTION
+    : synthesisRevision === 2 ? SYNTHESIS_DECISION_INSTRUCTION : SYNTHESIS_INSTRUCTION;
+  const method = synthesis ? synthesisRevision === undefined ? SYNTHESIS_METHOD
+    : `research-episodes-synthesis-v${synthesisRevision}` : TRAJECTORY_METHOD;
   if (config.output_language !== undefined && config.output_language !== 'zh' && config.output_language !== 'en') {
     throw new EffectRuntimeRequestError('output_language must be zh or en');
   }
@@ -247,7 +256,7 @@ export function prepareTrajectoryAssessment(params: Record<string, unknown>) {
   // Omission preserves the identity of already registered, language-unspecified artifacts.
   const configIdentity = {model: str(config.model, 'model'), effort: str(config.effort, 'effort'), service_tier: str(config.service_tier, 'service_tier'),
     ...(language ? {output_language: language} : {}), ...(synthesis ? {research_synthesis: true} : {}),
-    ...(synthesisRevision === 2 ? {research_synthesis_revision: synthesisRevision} : {})};
+    ...(synthesisRevision !== undefined ? {research_synthesis_revision: synthesisRevision} : {})};
   const numberedSources = input.sources.map(s => ({...s, text: s.text.split('\n').map((line, n) => `${s.first_line+n}: ${line}`).join('\n')}));
   const languageInstruction = language ? `\n\nOutput language: ${language === 'zh' ? 'Simplified Chinese (简体中文)' : 'English'}. Write every reason, limitation and episode question in this language, regardless of the evidence language.
 Return episode_questions exactly once per supplied episode: episode_id unchanged and question as a concise faithful rendering of its research question. Preserve uncertainty, negation, comparisons, numbers and units. Do not add conclusions to questions. These are analytical renderings, not replacements for original records.

tests/capabilities/test_trajectory_assessment_store.py

@@ -737,8 +737,35 @@ def test_revised_producer_preserves_frozen_legacy_job_and_requires_new_execution
     assert new["result"]["metrics"] == old["result"]["metrics"]
     assert inspect_pack(**scope, pack_id=legacy["pack_id"]) == old
     assert all(path.read_bytes() == content for path, content in before.items())
-    # Both frozen versions still use the same literal, source-qualified reader.
-    for op in [legacy_operation, operation]:
+    retained = {path: path.read_bytes() for path in scope["runtime_root"].glob("**/jobs/*/*") if path.is_file()}
+    selection = register_pack(**scope, input_path=input_path, source_root=sources,
+        title="Case-selection analysis", config={**current["config"], "research_synthesis_revision": 3},
+        base_pack_id=current["pack_id"])
+    selection_operation = {**scope, "pack_id": selection["pack_id"], "analysis_key": selection["analysis_key"]}
+    assert selection["analysis_key"] != current["analysis_key"]
+    assert inspect_pack(**scope, pack_id=selection["pack_id"])["job"] is None
+    execute_pack(**selection_operation, codex=provider)
+    selected = inspect_pack(**scope, pack_id=selection["pack_id"])
+    assert selected["result"]["method_version"] == "research-episodes-synthesis-v3"
+    assert selected["result"]["metrics"] == new["result"]["metrics"]
+    assert inspect_pack(**scope, pack_id=current["pack_id"]) == new
+    assert all(path.read_bytes() == content for path, content in retained.items())
+    retained = {path: path.read_bytes() for path in scope["runtime_root"].glob("**/jobs/*/*") if path.is_file()}
+    decision = register_pack(**scope, input_path=input_path, source_root=sources,
+        title="Decision-scoped analysis", config={**current["config"], "research_synthesis_revision": 4},
+        base_pack_id=selection["pack_id"])
+    decision_operation = {**scope, "pack_id": decision["pack_id"], "analysis_key": decision["analysis_key"]}
+    assert decision["analysis_key"] != selection["analysis_key"]
+    pending = inspect_pack(**scope, pack_id=decision["pack_id"])
+    assert pending["job"] is None and pending["result"] is None
+    execute_pack(**decision_operation, codex=provider)
+    decided = inspect_pack(**scope, pack_id=decision["pack_id"])
+    assert decided["result"]["method_version"] == "research-episodes-synthesis-v4"
+    assert decided["result"]["metrics"] == selected["result"]["metrics"]
+    assert inspect_pack(**scope, pack_id=selection["pack_id"]) == selected
+    assert all(path.read_bytes() == content for path, content in retained.items())
+    # Every version uses the same literal, source-qualified reader.
+    for op in [legacy_operation, operation, selection_operation, decision_operation]:
         assert read_citation(**op, source_id="source", first_line=2, last_line=2)["text"] == (
             "The native measurement did not improve utility · 原始测量。")
 

tests/control_plane_ts/trajectory_assessment.test.ts

@@ -533,12 +533,12 @@ test('artifact digests retain bare lowercase syntax, original bytes and error se
   }
 });
 
-for (const revision of [undefined, 2]) test(`research synthesis revision ${revision ?? 1} preserves episode citation boundaries`, () => {
+for (const revision of [undefined, 2, 3, 4]) test(`research synthesis revision ${revision ?? 1} preserves episode citation boundaries`, () => {
   const x = input();
   x.sources.push({id: 'other', artifact: 'other.txt', artifact_sha256: 'b'.repeat(64), first_line: 4, text: 'A competing explanation remains open'});
   x.episodes[1]!.source_ids = ['other'];
   const p = prepareTrajectoryAssessment({input: x, config: {...config, research_synthesis: true, output_language: 'zh',
-    ...(revision === 2 ? {research_synthesis_revision: revision} : {})}});
+    ...(revision !== undefined ? {research_synthesis_revision: revision} : {})}});
   const citation = {source_id: 's', first_line: 10, last_line: 10};
   const synthesis = {
     summary: {text: 'Negative evidence narrowed the question; the mechanism remains unresolved.', episode_ids: ['e1'], citations: [citation]},
@@ -590,6 +590,7 @@ test('synthesis revision separates new guidance while frozen and disabled reques
   assert.equal(prepared().analysis_key, 'bafc1cd4c93f25a274a6f6789eda79f3a1f9e197ff21a651bfd7f677f36cb607');
   assert.equal(legacy.analysis_key, '96c07478771c8bdb4db5e79999a31a25f8acc86cbc182bc8cb09ad52ac8a7a91');
   const next = prepareTrajectoryAssessment({input: input(), config: {...legacy.config, research_synthesis_revision: 2}});
+  assert.equal(next.analysis_key, '2d68cdd187eb5961ce022a39b8a68a67a363b6d376d3523381397febf1b935e0');
   assert.equal(next.method_version, 'research-episodes-synthesis-v2');
   assert.notEqual(next.analysis_key, legacy.analysis_key);
   assert.deepEqual(next.input, legacy.input);
@@ -602,10 +603,37 @@ test('synthesis revision separates new guidance while frozen and disabled reques
   assert.equal(compareTrajectoryAssessments({...next, previous: {...legacy, output: raw}}).can_reuse, false,
     'A guidance revision needs an explicit new analysis; do not relabel a saved judgment');
   assert.deepEqual(prepareTrajectoryAssessment({input: input(), config: legacy.config}), legacy);
-  for (const revision of [null, true, '2', 0, 1, 3]) {
+  const selection = prepareTrajectoryAssessment({input: input(), config: {...next.config, research_synthesis_revision: 3}});
+  assert.equal(selection.method_version, 'research-episodes-synthesis-v3');
+  assert.equal(selection.analysis_key, 'e348081086ca231be5aa9f22e3d695e66306410053afaaff3e9d2a5719cef033');
+  assert.notEqual(selection.analysis_key, next.analysis_key);
+  assert.deepEqual(selection.input, next.input);
+  assert.deepEqual(selection.output_schema, next.output_schema, 'Selection guidance reuses the report fields and evidence gates');
+  assert.equal(selection.prompt.split('\n\nEvidence pack:\n')[1], next.prompt.split('\n\nEvidence pack:\n')[1]);
+  assert.deepEqual(qualifyTrajectoryAssessment({...selection, output: raw}).metrics,
+    qualifyTrajectoryAssessment({...next, output: raw}).metrics);
+  assert.equal(compareTrajectoryAssessments({...selection, previous: {...next, output: raw}}).can_reuse, false);
+  assert.deepEqual(prepareTrajectoryAssessment({input: input(), config: next.config}), next,
+    'Explicit case-selection guidance must not upgrade a frozen revision 2 request');
+  const decision = prepareTrajectoryAssessment({input: input(), config: {...next.config, research_synthesis_revision: 4}});
+  assert.equal(decision.method_version, 'research-episodes-synthesis-v4');
+  assert.notEqual(decision.analysis_key, selection.analysis_key);
+  assert.deepEqual(decision.input, selection.input);
+  assert.deepEqual(decision.output_schema, selection.output_schema, 'Decision guidance adds no fields or scientific gate');
+  assert.equal(decision.prompt.split('\n\nEvidence pack:\n')[1], selection.prompt.split('\n\nEvidence pack:\n')[1]);
+  assert.deepEqual(qualifyTrajectoryAssessment({...decision, output: raw}).metrics,
+    qualifyTrajectoryAssessment({...selection, output: raw}).metrics);
+  assert.equal(compareTrajectoryAssessments({...decision, previous: {...selection, output: raw}}).can_reuse, false);
+  assert.deepEqual(prepareTrajectoryAssessment({input: input(), config: selection.config}), selection,
+    'A saved revision 3 report remains bound to its original instructions and job');
+  for (const revision of [null, true, '2', 0, 1, 5]) {
     assert.throws(() => prepareTrajectoryAssessment({input: input(), config: {...config,
       research_synthesis: true, research_synthesis_revision: revision}}), /supported revision/);
   }
   assert.throws(() => prepareTrajectoryAssessment({input: input(), config: {...config,
     research_synthesis: false, research_synthesis_revision: 2}}), /enabled synthesis/);
+  assert.throws(() => prepareTrajectoryAssessment({input: input(), config: {...config,
+    research_synthesis: false, research_synthesis_revision: 3}}), /enabled synthesis/);
+  assert.throws(() => prepareTrajectoryAssessment({input: input(), config: {...config,
+    research_synthesis: false, research_synthesis_revision: 4}}), /enabled synthesis/);
 });