Repository navigation
Conversation
There was a problem hiding this comment.
This review found 1 issue(s), including 1 blocking issue(s).
There was a problem hiding this comment.
This review found 1 issue(s), including 1 blocking issue(s).
There was a problem hiding this comment.
✅ Re-review complete · d6f3665
No findings in this incremental review.
lanyue-llk
left a comment
There was a problem hiding this comment.
Completed the P0/P1-only review of this companion to dataset PR12 at c2b468c. No confirmed P0/P1 in the reviewed source qualification, normal certification, retained receipt or assembly boundaries.
All 59 revised live-body cases select the new pre/post qualification path. Normal-entry controls reject empty or rewritten pre-call sources before provider execution, reject invalid post-call sources before certification, and reject missing, stale or altered qualification evidence. Swapping receipts between independently certified runs was also rejected. The previous zero-source-check path for these 59 cases is closed; the frozen body reference and acceptance thresholds remain intact.
Relevant lifecycle/certification/artifact checks passed 170 tests and 35 subtests. The 15 live-body qualification tests also passed with the current dataset package. These are controlled source/provider fixtures, not a fresh 550-URL source capture or measured provider cohort. No repository edits, paid calls, publication or merge.
|
Closing unmerged at the requester’s explicit direction. The 59 document cases will use their already reviewed frozen-reference body matching (five regions, each requiring 80%) without an additional pre/post live-page qualification requirement. PR lexmount/lexbench-webfetch#12 removes the added live_body declaration while preserving reference bytes, all body rules and existing facts/anchors. This execution companion is therefore no longer required. This does not assert that live pages are immutable; future drift remains a limitation of frozen-reference scoring. |
Live URLs evaluated against frozen body text could bypass current-source checks, so a rewritten official page could be scored as a provider failure and still enter a certified cohort. Add an explicit
source_verification.reference_mode: live_bodymode through the existing capture, freezer, preflight, postcheck and certificate validation paths.The independent current source must satisfy the unchanged body contract before and after calls. Preserve the original reference, regions and thresholds; retain a hash-bound qualification file outside Git. Missing, stale, altered or source-invalid evidence rejects certification. Minor changes within the declared coverage tolerance remain eligible. Existing
frozenandlive_exactbehavior is preserved.Companion to lexmount/lexbench-webfetch#12 and its review comment lexmount/lexbench-webfetch#12 (comment). That PR explicitly marks its 59 newly required live-body cases; older consumers reject the unknown mode rather than certify with zero checks. This PR is based on current main and does not include PR #65's provider changes. No release, dependency pin, provider request, historical result, or frozen reference is changed.
Qualification receipts are addressed by their content hash. Combining independently certified batches retains every selected receipt, even when their current pages differ within tolerance. The canonical frozen reference and scoring contract remain identical; capture-specific hashes may differ only through the existing equivalent-capture path.
Validation:
make package-checkrerun on stable final head 8a68908 passed.make package-checkrerun passed. Remote CI on d6f36659 passed, includingmake ciand rendered reference capture. The latest bot review passed and all review threads are resolved. Companion dataset PR Add isolated public WebArena fixed-trajectory replay #12 CI on c2b468c also passed. No live provider calls, historical rescoring, report publication, or merge.