From 6d45a91b8a89d2a4e11de5644981be902db71bc7 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Tue, 27 Oct 2026 21:18:24 +0800 Subject: [PATCH 1/7] docs(search): link the published benchmark release --- README.md | 2 +- docs/README.zh-CN.md | 2 +- docs/benchmarks.md | 6 +++--- docs/dependencies.md | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1d0ab38ad..34dff013b 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Choose a workflow for its task contract, setup and runnable example. | Track | What it tests | Benchmarks and task suites | Guide | | --- | --- | --- | --- | | Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | -| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch): scenario queries and a SimpleQA Verified retrieval adaptation | [Local example](examples/web_search/README.md) · [Full guide](harness/web_search/README.md) | +| Web Search | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search 1.1.0](https://github.com/lexmount/lexbench-websearch/releases/tag/bench-v1.1.0): scenario queries and a SimpleQA Verified retrieval adaptation | [Local example](examples/web_search/README.md) · [Full guide](harness/web_search/README.md) | | Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/README.md), adapted from [Lexbench-Headless-Browser scenario designs](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | | Browser Use | Let an Agent use different browsers to complete tasks | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified), using the [pinned evaluator fork](https://github.com/lexmount/webarena-verified/tree/lex-main) and [declared task selection](evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](examples/browser_use/README.md) | diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 92d59f44a..7f7a149b5 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -34,7 +34,7 @@ | 流程 | 测什么 | 对应评测集与任务集 | 使用指南 | | --- | --- | --- | --- | | Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | -| Web Search(预览) | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch):场景问题,以及用于检索评测的 SimpleQA Verified 改编任务 | [本地示例](../examples/web_search/README.md) · [完整指南](../harness/web_search/README.md) | +| Web Search | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search 1.1.0](https://github.com/lexmount/lexbench-websearch/releases/tag/bench-v1.1.0):场景问题,以及用于检索评测的 SimpleQA Verified 改编任务 | [本地示例](../examples/web_search/README.md) · [完整指南](../harness/web_search/README.md) | | 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/README.md),改编自 [Lexbench-Headless-Browser 场景设计](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios);[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json),由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | | Browser Use | 让 Agent 使用不同浏览器完成任务 | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified),使用[固定版本的评测器分支](https://github.com/lexmount/webarena-verified/tree/lex-main)和[明确选定的任务](../evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](../examples/browser_use/README.md) | diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 0cc509f75..13e030377 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -268,10 +268,10 @@ or renaming a participant requires data changes, without another renderer entry. ## Web Search integration -LexBench Web Search is installed from the immutable commit in `pyproject.toml` and `uv.lock`. Its task files and scoring code have one owner in that repository. Browser Eval provides providers, model/human annotation collection, latency/billing collection, audits and report publication. Report-specific execution profiles and receipts live with external evidence, selected explicitly by the caller. No sibling checkout is required. +Browser Eval uses [LexBench Web Search benchmark 1.1.0](https://github.com/lexmount/lexbench-websearch/releases/tag/bench-v1.1.0), installed as tooling package 0.1.1 from commit [`f07a2e2ddf1c1c511cb3944659534301c86ff9c9`](https://github.com/lexmount/lexbench-websearch/commit/f07a2e2ddf1c1c511cb3944659534301c86ff9c9). Both `pyproject.toml` and `uv.lock` pin this commit. Its task files and scoring code have one owner in that repository. Browser Eval provides providers, model/human annotation collection, latency/billing collection, audits and report publication. Report-specific execution profiles and receipts live with external evidence, selected explicitly by the caller. No sibling checkout is required. The pinned benchmark's `version` command checks its catalog and scoring state. Select task suites through the installed registry's `current` mapping; the bench edition fixes their snapshots. The harness resolves explicit dataset/field/hash references before freezing a run. Resolving the split historical profiles reproduces the original profile bytes and hashes, so saved calls keep their original identity. Model judgment evidence and measured results stay outside both repositories. -Change search queries, references or scoring in LexBench Web Search and pin the resulting commit here. Change provider behavior or report generation in Browser Eval. See the pinned benchmark's [data provenance](https://github.com/lexmount/lexbench-websearch/blob/a6d7b92987e32ef52563ee68bac4343db5686d95/docs/data.md) and [evaluation protocol](https://github.com/lexmount/lexbench-websearch/blob/a6d7b92987e32ef52563ee68bac4343db5686d95/docs/evaluation.md). +Change search queries, references or scoring in LexBench Web Search and pin the resulting commit here. Change provider behavior or report generation in Browser Eval. See the pinned benchmark's [data provenance](https://github.com/lexmount/lexbench-websearch/blob/f07a2e2ddf1c1c511cb3944659534301c86ff9c9/docs/data.md) and [evaluation protocol](https://github.com/lexmount/lexbench-websearch/blob/f07a2e2ddf1c1c511cb3944659534301c86ff9c9/docs/evaluation.md). -Update a merged Search benchmark revision with `make bump-lexbench BENCHMARK=lexbench-websearch REV=`. The shared updater checks upstream main ancestry and updates the package, lock and dataset-hash binding together. The initial extraction PR depends on the companion benchmark PR landing first. +Update a merged Search benchmark revision with `make bump-lexbench BENCHMARK=lexbench-websearch REV=`. The shared updater checks upstream main ancestry and updates the package, lock and dataset-hash binding together. diff --git a/docs/dependencies.md b/docs/dependencies.md index 459ef14b0..74aaa02b4 100644 --- a/docs/dependencies.md +++ b/docs/dependencies.md @@ -15,7 +15,7 @@ Use the declared Git source for the evaluator. A registry package with the same | Dependency | Role | Version authority | | --- | --- | --- | | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Task model, catalogs, deterministic acceptance scoring and frozen input validation | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md) | -| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Query definitions, provenance, source qualifications and retrieval scoring | Immutable Git revision in `pyproject.toml` and `uv.lock`; [Search guide](../harness/web_search/README.md) | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Query definitions, provenance, source qualifications and retrieval scoring | [Benchmark 1.1.0](https://github.com/lexmount/lexbench-websearch/releases/tag/bench-v1.1.0), tooling package 0.1.1; commit [`f07a2e2ddf1c1c511cb3944659534301c86ff9c9`](https://github.com/lexmount/lexbench-websearch/commit/f07a2e2ddf1c1c511cb3944659534301c86ff9c9) in `pyproject.toml` and `uv.lock`; [Search guide](../harness/web_search/README.md) | | [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) | HTML parsing | `pyproject.toml` and `uv.lock` | | [FastCDC](https://github.com/iscc/fastcdc-py) | Content chunking | `pyproject.toml` and `uv.lock` | From e2ed841896f33147c3bf4d8618f0a9ba8511b141 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Tue, 27 Oct 2026 21:23:15 +0800 Subject: [PATCH 2/7] docs(search): keep version details in dependency guides --- README.md | 2 +- docs/README.zh-CN.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 34dff013b..d53859b78 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Choose a workflow for its task contract, setup and runnable example. | Track | What it tests | Benchmarks and task suites | Guide | | --- | --- | --- | --- | | Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | -| Web Search | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search 1.1.0](https://github.com/lexmount/lexbench-websearch/releases/tag/bench-v1.1.0): scenario queries and a SimpleQA Verified retrieval adaptation | [Local example](examples/web_search/README.md) · [Full guide](harness/web_search/README.md) | +| Web Search | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch): scenario queries and a SimpleQA Verified retrieval adaptation | [Local example](examples/web_search/README.md) · [Full guide](harness/web_search/README.md) | | Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/README.md), adapted from [Lexbench-Headless-Browser scenario designs](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | | Browser Use | Let an Agent use different browsers to complete tasks | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified), using the [pinned evaluator fork](https://github.com/lexmount/webarena-verified/tree/lex-main) and [declared task selection](evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](examples/browser_use/README.md) | diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 7f7a149b5..c6efbea9a 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -34,7 +34,7 @@ | 流程 | 测什么 | 对应评测集与任务集 | 使用指南 | | --- | --- | --- | --- | | Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | -| Web Search | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search 1.1.0](https://github.com/lexmount/lexbench-websearch/releases/tag/bench-v1.1.0):场景问题,以及用于检索评测的 SimpleQA Verified 改编任务 | [本地示例](../examples/web_search/README.md) · [完整指南](../harness/web_search/README.md) | +| Web Search | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch):场景问题,以及用于检索评测的 SimpleQA Verified 改编任务 | [本地示例](../examples/web_search/README.md) · [完整指南](../harness/web_search/README.md) | | 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/README.md),改编自 [Lexbench-Headless-Browser 场景设计](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios);[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json),由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | | Browser Use | 让 Agent 使用不同浏览器完成任务 | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified),使用[固定版本的评测器分支](https://github.com/lexmount/webarena-verified/tree/lex-main)和[明确选定的任务](../evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](../examples/browser_use/README.md) | From a25aa75c18a6446655ba56ef5156dc8ee4db38d9 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Wed, 28 Oct 2026 10:42:44 +0800 Subject: [PATCH 3/7] docs: simplify Artifact Site links by language --- README.md | 4 ++-- docs/README.zh-CN.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index d53859b78..3b63e720b 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ Next, [try HTTP, Moli or Chromium on a local article](examples/web_fetch/README. ## Reports -To create and share a report, open Artifact Site ([Mainland China](https://yours.lexmount.cn/artifact-site) · [Global](https://yours.lexmount.com/artifact-site)) and install the skill offered there. Follow the skill to create and publish your report, then choose who can access it. +To create and share a report, open [Artifact Site](https://yours.lexmount.com/artifact-site) and install the skill offered there. Follow the skill to create and publish your report, then choose who can access it. Reports include comparisons, task results, full responses and failure evidence. Runs are saved under `~/.browser-eval/runs/`; use `--output /absolute/path/new-run` to choose another directory. @@ -84,7 +84,7 @@ Published versions, changes and dependency snapshots are listed in [GitHub Relea | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Versioned search queries, source references and retrieval scoring | | [Moli](https://github.com/lexmount/moli) | Browser evaluated through the same task contracts as other participants | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | CLI client for the Lexmount WebFetch service; separate from the evaluation harness | -| Artifact Site ([Mainland China](https://yours.lexmount.cn/artifact-site) · [Global](https://yours.lexmount.com/artifact-site)) | Report creation and sharing with access controls | +| [Artifact Site](https://yours.lexmount.com/artifact-site) | Report creation and sharing with access controls | | [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | Scenario design source for curated replay; separate protocol, driver and web-platform benchmarks | | [WebMainBench fork](https://github.com/lexmount/WebMainBench/tree/lex-main) · [upstream](https://github.com/opendatalab/WebMainBench) | Frozen-HTML extraction tasks and declared scoring corrections | | [WebArena-Verified fork](https://github.com/lexmount/webarena-verified/tree/lex-main) · [upstream](https://github.com/ServiceNow/webarena-verified) | Tasks and evaluator for Browser Use and WebArena trajectory replay | diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index c6efbea9a..436112f5d 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -56,7 +56,7 @@ uv run --frozen browser-harness-eval run --delay 0 ## 报告 -创建和分享报告,请打开 Artifact Site([中国大陆](https://yours.lexmount.cn/artifact-site) · [全球](https://yours.lexmount.com/artifact-site)),安装网站提供的 skill,按引导创建、发布报告,并设置分享权限。 +创建和分享报告,请打开 Artifact Site([.cn](https://yours.lexmount.cn/artifact-site) · [.com](https://yours.lexmount.com/artifact-site)),安装网站提供的 skill,按引导创建、发布报告,并设置分享权限。 报告包含结果对比、逐题结果、完整响应和失败证据。运行结果默认保存在 `~/.browser-eval/runs/`,也可用 `--output /absolute/path/new-run` 指定新目录。 @@ -84,7 +84,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | 搜索问题、来源参考和检索评分 | | [Moli](https://github.com/lexmount/moli) | 与其他参与者按相同任务规则评测的浏览器 | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 | -| Artifact Site([中国大陆](https://yours.lexmount.cn/artifact-site) · [全球](https://yours.lexmount.com/artifact-site)) | 报告创建、分享和权限控制 | +| Artifact Site([.cn](https://yours.lexmount.cn/artifact-site) · [.com](https://yours.lexmount.com/artifact-site)) | 报告创建、分享和权限控制 | | [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 自建轨迹的场景设计来源;另有独立的协议、自动化驱动和 Web 平台评测 | | [WebMainBench 分支](https://github.com/lexmount/WebMainBench/tree/lex-main) · [上游](https://github.com/opendatalab/WebMainBench) | 冻结 HTML 提取任务与有记录的评分修正 | | [WebArena-Verified 分支](https://github.com/lexmount/webarena-verified/tree/lex-main) · [上游](https://github.com/ServiceNow/webarena-verified) | Browser Use 和 WebArena 轨迹重放的题目与评测器 | From 81253ca42ba862508144c2d56d3dc47d17e708c3 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Wed, 28 Oct 2026 12:01:41 +0800 Subject: [PATCH 4/7] docs: use code bindings as benchmark version authority --- AGENTS.md | 2 ++ docs/benchmarks.md | 33 ++++++++++++++++++++++++++++----- docs/dependencies.md | 6 +++--- tests/test_release_metadata.py | 22 ++++++++++++++++++++++ 4 files changed, 55 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0ebd81c78..9f075ecdc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,6 +4,8 @@ Browser Eval owns execution adapters, evidence handling and report tooling. `lexmount/lexbench-webfetch` owns Web Fetch task populations, task models, deterministic acceptance scoring, compilation and frozen definition files. Import its pinned package directly; do not duplicate or re-export retired evaluator modules here. Resolve frozen historical catalogs/manifests through `lexbench_webfetch.inputs.definition_path`; keep experiment profiles and compact hash bindings in `evaluation/`. WebMainBench, WCXB and WebArena retain their own upstream task authorities. +Every benchmark track must record full commits for Git sources and exact task/reference/scorer hashes in its existing dependency or run bindings. Release tags are labels that resolve to recorded commits. Current-use documentation links to those bindings rather than duplicating selected version numbers; preserve immutable historical experiment and source citations. See `docs/benchmarks.md#benchmark-editions`. + # Report maintenance Repository documentation, identifiers, comments and docstrings use English. Report UI messages belong in language catalogs; source quotations and acceptance fixtures retain their original language. Keep the root README and `docs/README.zh-CN.md` aligned; additional README translations are welcome, but other guides need no parallel translation. Preserve original-language source quotations and hash-bound acceptance/evidence artifacts. Report-generation languages follow the track contracts below. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 13e030377..71d0c8d8f 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -56,9 +56,32 @@ The curated [Browser Trajectory Replay suite](../examples/trajectory_replay/READ ## Benchmark editions -Use a benchmark version together with exact task/reference and scoring-source -hashes. Keep the package's full Git commit pinned separately. A package update -does not select a new dataset for an existing report. +Every track identifies its benchmark code by the full source commit. A release +tag may provide a readable label, but the selected tag must resolve to a recorded +commit. Freeze task, reference and scoring-file hashes as well; a package version +or a moving branch name alone does not identify the evaluation. + +Use the existing owner for each selection: + +| Benchmark | Version authority | +| --- | --- | +| LexBench Web Fetch / Web Search | Git dependencies in `pyproject.toml`, resolved commits in `uv.lock`, and `evaluation/lexbench-*.json` | +| WebMainBench | Selected run profile's scorer commit and frozen dataset/scorer hashes | +| WCXB | Selected source manifest's `source_commit` and input hashes | +| WebArena for Browser Use and replay | Pinned evaluator dependency, runtime source verification and the selected task/trajectory bindings | +| Repository-curated replay | Browser Eval source commit and selected task/trajectory hashes | + +Read and validate the selected binding before execution; retain it with the run +and report. For data distributed as files, preserve content hashes even when no +upstream commit is available. Do not invent a source commit for an old snapshot. + +Current-use documentation links to these owners instead of repeating selected +versions or commits. Upstream `main` links in guides are navigation, not execution +inputs. Historical experiments and source attributions keep their immutable +citations; those records must not follow a newer dependency selection. + +Updating the installed benchmark does not select a new dataset for an existing +report. Create `bench-vMAJOR.MINOR.PATCH` tags only in the benchmark repository. Browser Eval publishes software versions as `vMAJOR.MINOR.PATCH` and records the @@ -268,10 +291,10 @@ or renaming a participant requires data changes, without another renderer entry. ## Web Search integration -Browser Eval uses [LexBench Web Search benchmark 1.1.0](https://github.com/lexmount/lexbench-websearch/releases/tag/bench-v1.1.0), installed as tooling package 0.1.1 from commit [`f07a2e2ddf1c1c511cb3944659534301c86ff9c9`](https://github.com/lexmount/lexbench-websearch/commit/f07a2e2ddf1c1c511cb3944659534301c86ff9c9). Both `pyproject.toml` and `uv.lock` pin this commit. Its task files and scoring code have one owner in that repository. Browser Eval provides providers, model/human annotation collection, latency/billing collection, audits and report publication. Report-specific execution profiles and receipts live with external evidence, selected explicitly by the caller. No sibling checkout is required. +LexBench Web Search is installed from the commit pinned in [pyproject.toml](../pyproject.toml), [uv.lock](../uv.lock) and the [definition binding](../evaluation/lexbench-websearch.json). Its task files and scoring code have one owner in that repository. Browser Eval provides providers, model/human annotation collection, latency/billing collection, audits and report publication. Report-specific execution profiles and receipts live with external evidence, selected explicitly by the caller. No sibling checkout is required. The pinned benchmark's `version` command checks its catalog and scoring state. Select task suites through the installed registry's `current` mapping; the bench edition fixes their snapshots. The harness resolves explicit dataset/field/hash references before freezing a run. Resolving the split historical profiles reproduces the original profile bytes and hashes, so saved calls keep their original identity. Model judgment evidence and measured results stay outside both repositories. -Change search queries, references or scoring in LexBench Web Search and pin the resulting commit here. Change provider behavior or report generation in Browser Eval. See the pinned benchmark's [data provenance](https://github.com/lexmount/lexbench-websearch/blob/f07a2e2ddf1c1c511cb3944659534301c86ff9c9/docs/data.md) and [evaluation protocol](https://github.com/lexmount/lexbench-websearch/blob/f07a2e2ddf1c1c511cb3944659534301c86ff9c9/docs/evaluation.md). +Change search queries, references or scoring in LexBench Web Search and pin the resulting commit here. Change provider behavior or report generation in Browser Eval. See the upstream [data provenance](https://github.com/lexmount/lexbench-websearch/blob/main/docs/data.md) and [evaluation protocol](https://github.com/lexmount/lexbench-websearch/blob/main/docs/evaluation.md). Update a merged Search benchmark revision with `make bump-lexbench BENCHMARK=lexbench-websearch REV=`. The shared updater checks upstream main ancestry and updates the package, lock and dataset-hash binding together. diff --git a/docs/dependencies.md b/docs/dependencies.md index 74aaa02b4..763d09245 100644 --- a/docs/dependencies.md +++ b/docs/dependencies.md @@ -8,14 +8,14 @@ Install the base CLI with `uv sync --frozen`. Add only the extras needed by your | [uv.lock](../uv.lock) | Resolved Python versions and hashes | | [package-lock.json](../package-lock.json) | Node dependency versions | -Use the declared Git source for the evaluator. A registry package with the same name and version may contain different code. +Use the declared source for each benchmark and evaluator. The full commit in the dependency lock or selected run binding is the version authority; a release tag is an optional label. See [benchmark identities](benchmarks.md#benchmark-editions). A registry package with the same name and version may contain different code. ## Core task and parsing dependencies | Dependency | Role | Version authority | | --- | --- | --- | | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Task model, catalogs, deterministic acceptance scoring and frozen input validation | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md) | -| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Query definitions, provenance, source qualifications and retrieval scoring | [Benchmark 1.1.0](https://github.com/lexmount/lexbench-websearch/releases/tag/bench-v1.1.0), tooling package 0.1.1; commit [`f07a2e2ddf1c1c511cb3944659534301c86ff9c9`](https://github.com/lexmount/lexbench-websearch/commit/f07a2e2ddf1c1c511cb3944659534301c86ff9c9) in `pyproject.toml` and `uv.lock`; [Search guide](../harness/web_search/README.md) | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Query definitions, provenance, source qualifications and retrieval scoring | Git revision in [pyproject.toml](../pyproject.toml), [uv.lock](../uv.lock) and the [definition binding](../evaluation/lexbench-websearch.json); [Search guide](../harness/web_search/README.md) | | [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) | HTML parsing | `pyproject.toml` and `uv.lock` | | [FastCDC](https://github.com/iscc/fastcdc-py) | Content chunking | `pyproject.toml` and `uv.lock` | @@ -23,7 +23,7 @@ Use the declared Git source for the evaluator. A registry package with the same | Dependency | Repository and relevant branch | Required source | | --- | --- | --- | -| WebArena-Verified evaluator | [lexmount/webarena-verified](https://github.com/lexmount/webarena-verified), maintained evaluator work on `lex-main` | [`da7b992eb10de78000f11a7f34fd4590c171fbd6`](https://github.com/lexmount/webarena-verified/commit/da7b992eb10de78000f11a7f34fd4590c171fbd6); installed by `browser-use-eval` from the same pin in pyproject and uv.lock | +| WebArena-Verified evaluator | [lexmount/webarena-verified](https://github.com/lexmount/webarena-verified) | Git dependency in [pyproject.toml](../pyproject.toml) and [uv.lock](../uv.lock); [runtime source check](../harness/trajectory_replay/webarena_runtime_source.py) | | CowAgent Web Fetch baseline | [zhayujie/CowAgent](https://github.com/zhayujie/CowAgent) | [`ed5bb344cfe42cee443c8266ba47ee666d8d7471`](https://github.com/zhayujie/CowAgent/commit/ed5bb344cfe42cee443c8266ba47ee666d8d7471) | | OpenClaw Web Fetch baseline | [openclaw/openclaw](https://github.com/openclaw/openclaw) | [`12c4bdf7482f24c85b011e9d858fdec3c245101c`](https://github.com/openclaw/openclaw/commit/12c4bdf7482f24c85b011e9d858fdec3c245101c) | | DeepSeek Harness Web Fetch baseline | [deepseek-ai/deepseek-harness](https://github.com/deepseek-ai/deepseek-harness) | [`dd6322d604e00eec1ba5e0c8541159906a21094a`](https://github.com/deepseek-ai/deepseek-harness/commit/dd6322d604e00eec1ba5e0c8541159906a21094a) | diff --git a/tests/test_release_metadata.py b/tests/test_release_metadata.py index 1db292561..607e506db 100644 --- a/tests/test_release_metadata.py +++ b/tests/test_release_metadata.py @@ -70,6 +70,28 @@ def test_declared_benchmark_pins_match_lock_and_installation(): assert origin["vcs_info"]["commit_id"] == revision, name +def test_all_git_dependencies_resolve_to_locked_source_commits(): + from urllib.parse import parse_qs, urlsplit + + project = source_project() + packages = tomllib.loads((ROOT / "uv.lock").read_text())["package"] + dependencies = list(project["dependencies"]) + for extra in project.get("optional-dependencies", {}).values(): + dependencies.extend(extra) + for dependency in dependencies: + if " @ git+" not in dependency: + continue + name, source = dependency.split(" @ git+", 1) + repository, requested = source.rsplit("@", 1) + package = next(p for p in packages if p["name"] == name) + locked = urlsplit(package["source"]["git"]) + assert re.fullmatch(r"[0-9a-f]{40}", locked.fragment), name + assert locked._replace(query="", fragment="").geturl() == repository, name + assert parse_qs(locked.query).get("rev") == [requested], name + if re.fullmatch(r"[0-9a-f]{40}", requested): + assert locked.fragment == requested, name + + def test_installed_benchmarks_match_their_frozen_catalogs(): from importlib import import_module From b09fb11fc9ba280f1ab7a6dd6b9aead79deeca06 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Wed, 28 Oct 2026 12:22:11 +0800 Subject: [PATCH 5/7] docs: reference installation code instead of repeating pins --- docs/dependencies.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/docs/dependencies.md b/docs/dependencies.md index 763d09245..254f2d90c 100644 --- a/docs/dependencies.md +++ b/docs/dependencies.md @@ -24,19 +24,17 @@ Use the declared source for each benchmark and evaluator. The full commit in the | Dependency | Repository and relevant branch | Required source | | --- | --- | --- | | WebArena-Verified evaluator | [lexmount/webarena-verified](https://github.com/lexmount/webarena-verified) | Git dependency in [pyproject.toml](../pyproject.toml) and [uv.lock](../uv.lock); [runtime source check](../harness/trajectory_replay/webarena_runtime_source.py) | -| CowAgent Web Fetch baseline | [zhayujie/CowAgent](https://github.com/zhayujie/CowAgent) | [`ed5bb344cfe42cee443c8266ba47ee666d8d7471`](https://github.com/zhayujie/CowAgent/commit/ed5bb344cfe42cee443c8266ba47ee666d8d7471) | -| OpenClaw Web Fetch baseline | [openclaw/openclaw](https://github.com/openclaw/openclaw) | [`12c4bdf7482f24c85b011e9d858fdec3c245101c`](https://github.com/openclaw/openclaw/commit/12c4bdf7482f24c85b011e9d858fdec3c245101c) | -| DeepSeek Harness Web Fetch baseline | [deepseek-ai/deepseek-harness](https://github.com/deepseek-ai/deepseek-harness) | [`dd6322d604e00eec1ba5e0c8541159906a21094a`](https://github.com/deepseek-ai/deepseek-harness/commit/dd6322d604e00eec1ba5e0c8541159906a21094a) | +| CowAgent Web Fetch baseline | [zhayujie/CowAgent](https://github.com/zhayujie/CowAgent) | `cowagent_rev` in [bootstrap_oss_baselines.sh](../scripts/bootstrap_oss_baselines.sh); configured `expected_repo_revision` | +| OpenClaw Web Fetch baseline | [openclaw/openclaw](https://github.com/openclaw/openclaw) | `openclaw_rev` in [bootstrap_oss_baselines.sh](../scripts/bootstrap_oss_baselines.sh); configured `expected_repo_revision` | +| DeepSeek Harness Web Fetch baseline | [deepseek-ai/deepseek-harness](https://github.com/deepseek-ai/deepseek-harness) | `dsh_rev` in [bootstrap_oss_baselines.sh](../scripts/bootstrap_oss_baselines.sh); configured `expected_repo_revision` | ### Install the historical Web Fetch baselines -[bootstrap_oss_baselines.sh](../scripts/bootstrap_oss_baselines.sh) checks out CowAgent, OpenClaw and DeepSeek Harness at the commits above and builds their tools. It rejects uncommitted changes and unexpected repository origins. The defaults match [providers.example.json](../configs/providers.example.json) and [providers.resources.json](../configs/providers.resources.json). +[bootstrap_oss_baselines.sh](../scripts/bootstrap_oss_baselines.sh) checks out CowAgent, OpenClaw and DeepSeek Harness at its declared commits and builds their tools. It rejects uncommitted changes and unexpected repository origins. The defaults match [providers.example.json](../configs/providers.example.json) and [providers.resources.json](../configs/providers.resources.json). -| Historical runtime | Version | -| --- | --- | -| Node.js | 24.15.0 | -| pnpm | 12.1.0 | -| Python | 3.11.12 | +Node.js, pnpm and Python installation versions are declared in that script. The +selected provider configuration checks them through its `expected_*_version` +fields. Read the script before running it: it downloads and builds dependencies, and replaces the checkout's `.venv`. Use a separate clone if you need to keep another environment. The Python patch pin applies to these baselines; the base CLI supports a wider range. @@ -52,7 +50,7 @@ Browser Use records its configured clean CowAgent checkout separately. It need n | Chromium / Playwright | [Playwright browser installation](https://playwright.dev/python/docs/browsers) | Playwright package in uv.lock and its matching browser, or an explicit configured binary | | Lightpanda | [lightpanda-io/browser releases](https://github.com/lightpanda-io/browser/releases) | Selected release bytes/version; a mutable nightly URL alone is insufficient for formal reproduction | | Obscura | [h4ckf0r0day/obscura releases](https://github.com/h4ckf0r0day/obscura/releases) | Selected provider identity and cohort evidence | -| HTML to Markdown | [node-html-markdown](https://github.com/crosstype/node-html-markdown) | `package-lock.json`; adapters require 2.0.0. Install with `npm ci` | +| HTML to Markdown | [node-html-markdown](https://github.com/crosstype/node-html-markdown) | [package-lock.json](../package-lock.json); install with `npm ci` | | LangChain loader / Crawl4AI | [LangChain Community](https://github.com/langchain-ai/langchain-community) · [Crawl4AI](https://github.com/unclecode/crawl4ai) | `oss-eval`, uv.lock and the selected method's interpreter/library pins | Prepare browser binaries, Docker images, site data, models and accounts separately. [`.env.example`](../.env.example) lists paths and credential names. `browser-harness-eval accounts` prints account setup links without making evaluation calls. Publishing uses separate [hosting credentials](report-hosting.md). @@ -61,10 +59,10 @@ Prepare browser binaries, Docker images, site data, models and accounts separate [WebMainBench](https://github.com/opendatalab/WebMainBench) supplies the frozen-HTML dataset and official scorer. The maintained [fork](https://github.com/lexmount/WebMainBench/tree/lex-main) contains declared scoring corrections: -| Scorer | Commit used by the full comparison | +| Scorer | Version authority | | --- | --- | -| Official | `9d991bdc00c57b57521499494d96be85c31317ba` | -| Corrected fork | `65b49124b923e59f3cfcd839ee1af4628c36f340` | +| Official default | `SCORER_COMMIT` in [webmainbench_scoring.py](../harness/web_fetch/webmainbench_scoring.py) | +| Selected correction | `scorer_commit` in the selected experiment's correction binding; [existing binding](../evaluation/webmainbench-full/scoring-correction-binding.json) | Choose the scorer named by the experiment's [profile and correction records](../evaluation/webmainbench-full/). See the [scoring guide](../harness/web_fetch/WEBMAINBENCH.md); switching to another branch tip changes the evaluation. From ba9f1c509f31bb87b3d9edc5901ec698e632b1f3 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Wed, 28 Oct 2026 12:24:52 +0800 Subject: [PATCH 6/7] docs: clarify Chinese Artifact Site entry labels --- docs/README.zh-CN.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 436112f5d..8f918ae73 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -56,7 +56,7 @@ uv run --frozen browser-harness-eval run --delay 0 ## 报告 -创建和分享报告,请打开 Artifact Site([.cn](https://yours.lexmount.cn/artifact-site) · [.com](https://yours.lexmount.com/artifact-site)),安装网站提供的 skill,按引导创建、发布报告,并设置分享权限。 +创建和分享报告,请打开 Artifact Site([中国大陆 .cn](https://yours.lexmount.cn/artifact-site) · [全球 .com](https://yours.lexmount.com/artifact-site)),安装网站提供的 skill,按引导创建、发布报告,并设置分享权限。 报告包含结果对比、逐题结果、完整响应和失败证据。运行结果默认保存在 `~/.browser-eval/runs/`,也可用 `--output /absolute/path/new-run` 指定新目录。 @@ -84,7 +84,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | 搜索问题、来源参考和检索评分 | | [Moli](https://github.com/lexmount/moli) | 与其他参与者按相同任务规则评测的浏览器 | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 | -| Artifact Site([.cn](https://yours.lexmount.cn/artifact-site) · [.com](https://yours.lexmount.com/artifact-site)) | 报告创建、分享和权限控制 | +| Artifact Site([中国大陆 .cn](https://yours.lexmount.cn/artifact-site) · [全球 .com](https://yours.lexmount.com/artifact-site)) | 报告创建、分享和权限控制 | | [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 自建轨迹的场景设计来源;另有独立的协议、自动化驱动和 Web 平台评测 | | [WebMainBench 分支](https://github.com/lexmount/WebMainBench/tree/lex-main) · [上游](https://github.com/opendatalab/WebMainBench) | 冻结 HTML 提取任务与有记录的评分修正 | | [WebArena-Verified 分支](https://github.com/lexmount/webarena-verified/tree/lex-main) · [上游](https://github.com/ServiceNow/webarena-verified) | Browser Use 和 WebArena 轨迹重放的题目与评测器 | From 3ecda974fd0453d91f3de4bbc4147ea52d29b407 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Wed, 28 Oct 2026 12:31:31 +0800 Subject: [PATCH 7/7] docs: list both Artifact Site entries in global-first order --- README.md | 4 ++-- docs/README.zh-CN.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 3b63e720b..14aa14942 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ Next, [try HTTP, Moli or Chromium on a local article](examples/web_fetch/README. ## Reports -To create and share a report, open [Artifact Site](https://yours.lexmount.com/artifact-site) and install the skill offered there. Follow the skill to create and publish your report, then choose who can access it. +To create and share a report, open Artifact Site ([Global .com](https://yours.lexmount.com/artifact-site) · [China .cn](https://yours.lexmount.cn/artifact-site)) and install the skill offered there. Follow the skill to create and publish your report, then choose who can access it. Reports include comparisons, task results, full responses and failure evidence. Runs are saved under `~/.browser-eval/runs/`; use `--output /absolute/path/new-run` to choose another directory. @@ -84,7 +84,7 @@ Published versions, changes and dependency snapshots are listed in [GitHub Relea | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Versioned search queries, source references and retrieval scoring | | [Moli](https://github.com/lexmount/moli) | Browser evaluated through the same task contracts as other participants | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | CLI client for the Lexmount WebFetch service; separate from the evaluation harness | -| [Artifact Site](https://yours.lexmount.com/artifact-site) | Report creation and sharing with access controls | +| Artifact Site ([Global .com](https://yours.lexmount.com/artifact-site) · [China .cn](https://yours.lexmount.cn/artifact-site)) | Report creation and sharing with access controls | | [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | Scenario design source for curated replay; separate protocol, driver and web-platform benchmarks | | [WebMainBench fork](https://github.com/lexmount/WebMainBench/tree/lex-main) · [upstream](https://github.com/opendatalab/WebMainBench) | Frozen-HTML extraction tasks and declared scoring corrections | | [WebArena-Verified fork](https://github.com/lexmount/webarena-verified/tree/lex-main) · [upstream](https://github.com/ServiceNow/webarena-verified) | Tasks and evaluator for Browser Use and WebArena trajectory replay | diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 8f918ae73..d130e47b0 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -56,7 +56,7 @@ uv run --frozen browser-harness-eval run --delay 0 ## 报告 -创建和分享报告,请打开 Artifact Site([中国大陆 .cn](https://yours.lexmount.cn/artifact-site) · [全球 .com](https://yours.lexmount.com/artifact-site)),安装网站提供的 skill,按引导创建、发布报告,并设置分享权限。 +创建和分享报告,请打开 Artifact Site([全球站 .com](https://yours.lexmount.com/artifact-site) · [中国站 .cn](https://yours.lexmount.cn/artifact-site)),安装网站提供的 skill,按引导创建、发布报告,并设置分享权限。 报告包含结果对比、逐题结果、完整响应和失败证据。运行结果默认保存在 `~/.browser-eval/runs/`,也可用 `--output /absolute/path/new-run` 指定新目录。 @@ -84,7 +84,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | 搜索问题、来源参考和检索评分 | | [Moli](https://github.com/lexmount/moli) | 与其他参与者按相同任务规则评测的浏览器 | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 | -| Artifact Site([中国大陆 .cn](https://yours.lexmount.cn/artifact-site) · [全球 .com](https://yours.lexmount.com/artifact-site)) | 报告创建、分享和权限控制 | +| Artifact Site([全球站 .com](https://yours.lexmount.com/artifact-site) · [中国站 .cn](https://yours.lexmount.cn/artifact-site)) | 报告创建、分享和权限控制 | | [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 自建轨迹的场景设计来源;另有独立的协议、自动化驱动和 Web 平台评测 | | [WebMainBench 分支](https://github.com/lexmount/WebMainBench/tree/lex-main) · [上游](https://github.com/opendatalab/WebMainBench) | 冻结 HTML 提取任务与有记录的评分修正 | | [WebArena-Verified 分支](https://github.com/lexmount/webarena-verified/tree/lex-main) · [上游](https://github.com/ServiceNow/webarena-verified) | Browser Use 和 WebArena 轨迹重放的题目与评测器 |