Skip to content

fix(webfetch): qualify current sources against frozen body rules - #75

Closed
Maisiqi wants to merge 3 commits into
mainfrom
codex/live-body-qualification
Closed

Maisiqi wants to merge 3 commits into
mainfrom
codex/live-body-qualification

Conversation

@Maisiqi

@Maisiqi Maisiqi commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Live URLs evaluated against frozen body text could bypass current-source checks, so a rewritten official page could be scored as a provider failure and still enter a certified cohort. Add an explicit source_verification.reference_mode: live_body mode through the existing capture, freezer, preflight, postcheck and certificate validation paths.

The independent current source must satisfy the unchanged body contract before and after calls. Preserve the original reference, regions and thresholds; retain a hash-bound qualification file outside Git. Missing, stale, altered or source-invalid evidence rejects certification. Minor changes within the declared coverage tolerance remain eligible. Existing frozen and live_exact behavior is preserved.

Companion to lexmount/lexbench-webfetch#12 and its review comment lexmount/lexbench-webfetch#12 (comment). That PR explicitly marks its 59 newly required live-body cases; older consumers reject the unknown mode rather than certify with zero checks. This PR is based on current main and does not include PR #65's provider changes. No release, dependency pin, provider request, historical result, or frozen reference is changed.

Qualification receipts are addressed by their content hash. Combining independently certified batches retains every selected receipt, even when their current pages differ within tolerance. The canonical frozen reference and scoring contract remain identical; capture-specific hashes may differ only through the existing equivalent-capture path.

Validation:

  • Full Python suite before the receipt follow-up: 7,045 passed, 132 skipped, 2,935 subtests passed. UI tests and fixture smoke passed in that run; package smoke was invalidated by source edits during the build. The full make package-check rerun on stable final head 8a68908 passed.
  • Final receipt follow-up: 207 lifecycle/assembly/revision tests and 50 subtests passed; lint passed. Includes two independently certified, slightly different sources combined through production assembly helpers, both certificates revalidated, and rejection of tampering or changed body thresholds.
  • Public certification-entry controls cover pre-call source rejection, post-call drift, tolerance-valid changes, preserved references, and missing/stale/altered qualification evidence. All captures and provider responses are synthetic fixtures.
  • The raw-byte receipt follow-up (d6f3665) rejects newline-only tampering during both preflight and certificate revalidation; its final make package-check rerun passed. Remote CI on d6f36659 passed, including make ci and rendered reference capture. The latest bot review passed and all review threads are resolved. Companion dataset PR Add isolated public WebArena fixed-trajectory replay #12 CI on c2b468c also passed. No live provider calls, historical rescoring, report publication, or merge.

@lexmount-ocr lexmount-ocr Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Review complete · 4a9570f

This review found 1 issue(s), including 1 blocking issue(s).

Comment thread harness/web_fetch/reference_lifecycle.py
@Maisiqi
Maisiqi marked this pull request as ready for review October 27, 2026 14:33

@lexmount-ocr lexmount-ocr Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Review complete · 8a68908

This review found 1 issue(s), including 1 blocking issue(s).

Comment thread harness/web_fetch/reference_lifecycle.py Outdated

@lexmount-ocr lexmount-ocr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Re-review complete · d6f3665

No findings in this incremental review.

@lanyue-llk lanyue-llk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed the P0/P1-only review of this companion to dataset PR12 at c2b468c. No confirmed P0/P1 in the reviewed source qualification, normal certification, retained receipt or assembly boundaries.

All 59 revised live-body cases select the new pre/post qualification path. Normal-entry controls reject empty or rewritten pre-call sources before provider execution, reject invalid post-call sources before certification, and reject missing, stale or altered qualification evidence. Swapping receipts between independently certified runs was also rejected. The previous zero-source-check path for these 59 cases is closed; the frozen body reference and acceptance thresholds remain intact.

Relevant lifecycle/certification/artifact checks passed 170 tests and 35 subtests. The 15 live-body qualification tests also passed with the current dataset package. These are controlled source/provider fixtures, not a fresh 550-URL source capture or measured provider cohort. No repository edits, paid calls, publication or merge.

@Maisiqi

Maisiqi commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Closing unmerged at the requester’s explicit direction. The 59 document cases will use their already reviewed frozen-reference body matching (five regions, each requiring 80%) without an additional pre/post live-page qualification requirement. PR lexmount/lexbench-webfetch#12 removes the added live_body declaration while preserving reference bytes, all body rules and existing facts/anchors. This execution companion is therefore no longer required. This does not assert that live pages are immutable; future drift remains a limitation of frozen-reference scoring.

@Maisiqi Maisiqi closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants