Calibrate INTERIM Perplexity-automation constants after Phase 4 aggregator collects ~20 runs

[intellegix]

Context

Three INTERIM constants in council-automation were tuned via single-anecdote extrapolation (the 2026-05-27 /extended-research audit verdict: STRUCTURAL-UNRESOLVABLE). They need empirical re-evaluation against real-world data, NOT another guess.

This issue lives outside the codebase so the inline # INTERIM PENDING PHASE 4 CALIBRATION comments don't silently rot if Phase 4 review slips.

INTERIM constants to calibrate

File	Constant	Current value	Original value
council-automation/extended_research_runner.py	ARTIFACT_TEXT_TOKEN_CAP	5000 (~20 KB)	8000 (~32 KB)
council-automation/council_browser.py	_is_reasoning_trail_only char threshold	2500	1500
council-automation/extended_research_runner.py	pre-flight oversize WARN	16 KB	18 KB

Kept-at-200 in this round (Step 8 critique reverted my proposed lowering): synthesis-mount wait_for_function threshold in council_browser.py.

Source-of-truth commits

• d90e4cf — Phase 1-3 of empirical reassessment (instrument + parser provisions + cap relaxation)
• 4c9ae9a — Phase 1 of follow-ups (keeper-core trio commit)
• Phase 2-3 of follow-ups (commit pending) — calibration aggregator + per-query instrumentation

Pre-state recovery anchor: tag pre-perplexity-reassessment-20260529.

Trigger condition

After Phase 1 instrumentation accumulates ≥20 entries in ~/.claude/council-automation/calibration-summary.jsonl from real /extended-research runs, evaluate:

```bash
python ~/.claude/council-automation/calibration_log.py 20
```

Decision matrix

Signal	Threshold	Action
pct_retry_fired	> 25%	Investigate parser-vs-Perplexity attribution (Phase 2+3 confound documented)
pct_parse_failed	> 10%	Likely needs Phase 5 prompt-architecture refactor
mean_raw_response_chars	< 1000	Caps may be too tight; consider raising ARTIFACT_TEXT_TOKEN_CAP
pct_partial_prefix_salvage	> 5%	Truncation issue is real; consider DECOMPOSE 1x → 1.25x (carefully)
chrome_path_used shows >5% local_* or cdp_busy	—	Keeper architecture has a regression to chase

Banned rollback combination

Per Step 8 critique Q1: if Phase 2 wrapper-extraction parser provisions in extract_json are ever reverted, the reasoning-trail threshold MUST also revert to ≤1500 in the same change. The 2500 threshold widens the zone Phase 2 covers; without Phase 2, 2500 leaks valid wrapper-format responses past the heuristic.

Phase 5 (prompt-architecture refactor) trigger

If after 20+ runs, INCOMPLETE_RESPONSE rate > 15% on small artifacts (< 10 KB) where caps cannot be the cause → trigger Phase 5 plan synthesis.

Definition of done

• Run python calibration_log.py 20 and paste the output here.
• Apply the decision-matrix recommendation OR document why no change is warranted.
• Update the INTERIM comments at the constant sites (either remove "INTERIM" qualifier if values confirmed, or update with the new tuned value + revision date).
• Close this issue.

[intellegix]

First empirical data — render-cliff is at ~18 KB, not 30 KB

Captured 2026-05-29 ~14:15 PT after another session triggered three /extended-research failures on a 12.9 KB artifact. Instrumentation from commits d90e4cf (per-pass) and 49fdaab (per-query) recorded 21 passes / 26 queries across this morning's and afternoon's runs. Not yet 20 distinct runs (decision-matrix threshold), but signal is sharp enough to log as a calibration data point.

Per-query instrumentation (council_browser.py)

100% chrome_path_used=cdp_attached across all 26 queries — keeper-CDP architecture working as designed, no fallback paths exercised. 100% exit_reason=completed, zero cdp_busy, zero submit_lock_failed.

The cliff (sorted by query_chars)

query_chars	extracted_synthesis_chars	outcome
14,515	7,991	✓
16,548	6,606	✓
17,959	6,819	✓
18,182	200	✗ partial-mount shell
19,687	220	✗
19,883	480	✗
24,443	203	✗

The cliff sits at ~18 KB total prompt size, not the 30 KB extrapolation from the single 65 KB anecdote that originally drove ARTIFACT_TEXT_TOKEN_CAP=8000 → 5000 → (TBD).

Calibration aggregator summary

# Calibration summary — last 2 runs (20 total passes)
- pct_retry_fired:           0.0%  (0/20)
- pct_partial_prefix_salvage: 0.0%  (0/20)
- pct_completed:             100.0%  (20/20)
- pct_parse_failed:          0.0%  (0/20)
- mean_prompt_chars:         17121
- mean_raw_response_chars:   22093

Note: pct_completed=100% but multiple verdict_hint=INCOMPLETE_RESPONSE — these are passes where _defensive_inject_required filled in missing fields from an effectively-empty response. The retry mechanism + partial-prefix salvage never fired (0% / 0%) because the failure mode is "tiny extracted body" (200-480 chars), not "truncated JSON fragment" — _extract_longest_valid_json_prefix doesn't help for partial-mount shells because there's no JSON to recover.

Decision-matrix evaluation

Against the matrix from the issue body:

Signal	Threshold	Observed	Action
pct_retry_fired	> 25%	0%	No action
pct_parse_failed	> 10%	0%	No action
mean_raw_response_chars	< 1000	22093	No action (response IS coming back)
pct_partial_prefix_salvage	> 5%	0%	No action
chrome_path_used local_*/cdp_busy	> 5%	0%	No action

The matrix doesn't currently capture the failure mode we're actually seeing: prompts above 18 KB produce a partial-mount shell (.prose text < 500 chars) that the runner records as status=COMPLETED + verdict_hint=INCOMPLETE_RESPONSE. Suggested matrix addition:

Signal	Threshold	Action
synth_extracted < 500 chars rate on queries with query_chars > 18000	> 30%	Lower ARTIFACT_TEXT_TOKEN_CAP to 4000 (~16 KB)

Provisional recommendation (NOT yet enacted)

When ≥ 20 distinct runs accumulate, the data is likely to support:

• ARTIFACT_TEXT_TOKEN_CAP: 5000 → 4000 (~16 KB target, stays well below empirical 18 KB cliff)
• Pre-flight WARN: 16 KB → 14 KB to match
• Reasoning-trail threshold: leave 2500 (partial-mount shells at 200-480 chars are caught by the wait_for_function('.prose innerText > 200') guard, not the reasoning-trail heuristic)

Not enacting yet — per the issue's "decision matrix" framing, single-day data on three runs isn't sufficient. Posting this as a checkpoint so the empirical pattern is preserved while data accumulates.

Reproducer artifact

The 12.9 KB E2E report at ~/projects/nl2sql-demo/.workflow/frontend-e2e-report.md reliably reproduces the partial-mount shell at PASS 1 DECOMPOSE on this date. Worth keeping as a regression-test artifact when Phase 4 calibration is eventually enacted.

[intellegix]

Day-1 adjustment enacted: ARTIFACT_TEXT_TOKEN_CAP reverted 5000 → 3500

Commit 7ee00ea. The Phase 4 trigger threshold (20-run minimum before adjustment) was designed for slow cumulative calibration; today's signal was loud enough on day 1 to enact the data-driven adjustment immediately.

Trigger justification (vs the issue body's 20-run threshold)

The issue body's matrix assumed gradual data accumulation. The empirical pattern observed today is qualitatively different from "slow cumulative drift":

• Sharp bimodal cluster, not a distribution — successful extractions cleanly cluster at 5-8 KB synthesis on prompts < 18 KB; failures cleanly cluster at 200-480 char partial-mount shells on prompts > 18 KB. No gray zone.
• Reproduced in 3 separate runs in another Claude Code session today, all failing identically (PASS 1 + PASS 2 INCOMPLETE, then auto-abort). The pattern is consistent enough that more data wouldn't change the verdict.
• Failure rate within the cliff zone is ~100% based on observed queries: every single query with query_chars > 18000 produced a < 500 char extraction in the captured data.

This is a "the data is the answer" case, not a "wait for confidence intervals to narrow" case. The 20-run threshold remains correct for future ambiguous-signal adjustments (e.g., reasoning-trail threshold tuning); it's just not the right gate for an already-unambiguous failure mode.

What the revert preserves

The bigger architectural moves from this morning's commits (d90e4cf + 49fdaab) all stay:

• Phase 1 instrumentation (the thing that surfaced this in 5 hours)
• Phase 2 wrapper-aware parser provisions in extract_json
• Phase 3's reasoning-trail threshold = 2500 (covered by Phase 2 parser per the documented banned rollback combo)
• Synthesis-mount threshold = 200 (Step 8 critique rationale)
• Pre-flight WARN at 16 KB (still triggers appropriately given new 3500 cap → ~19 KB max prompt)

Only the cap relaxation is reverted. The instrumentation infrastructure is exactly what made the early-revert defensible.

Provisional next-band recommendation

When 20+ runs accumulate, the data may support raising back to 4000 tokens (~16 KB artifact, ~21 KB prompt — at the cliff with smaller margin). Today's data doesn't yet support that; 3500 is what's empirically validated. Re-evaluation deferred to the original 20-run threshold for any cap > 3500.