Browser Eval is an extensible evaluation harness for browsers and web agents. It combines task contracts, adapters, saved evidence and scoring to produce inspectable reports.
The following workflows have runnable guides. Additional workflows, including Web Search, can use the same adapter and task-contract model; see contributing.
| Track | What it tests | Guide |
|---|---|---|
| Web Fetch | Retrieve the requested page content, records, resources or files | Local example · Full guide |
| Browser Trajectory Replay | Complete the same recorded browser actions across implementations | Curated suite · WebArena replay |
| Browser Use | Let an Agent use different browsers to complete tasks | CowAgent + WebArena |
Web Fetch supports local browsers, hosted services and extraction tools. See adapters for available methods and dependencies for installation requirements. Replay and Browser Use have separate runners and environments.
Task definitions and scoring come from the selected benchmark: LexBench Web Fetch, WebMainBench, WCXB or WebArena. See benchmark integration.
Install uv, then run the offline example:
git clone https://github.com/lexmount/browser-eval.git
cd browser-eval
uv sync --frozen
uv run --frozen browser-harness-eval validate
uv run --frozen browser-harness-eval run --delay 0Open the report path printed in the terminal. The example has one expected pass and one expected failure; it needs no browser, Docker or API key. Python 3.11–3.14 is supported; development uses 3.11.
Next, try HTTP, Moli or Chromium on a local article. Optional adapters, service credentials and live evaluation commands are covered in the track guides above.
Reports include comparisons, task results, full responses and failure evidence. Runs are saved under ~/.browser-eval/runs/; use --output /absolute/path/new-run to choose another directory.
Rebuild from saved evidence without repeating calls:
uv run --frozen browser-harness-eval report /absolute/path/to/runEnglish is the default; add --language zh when running or rebuilding for Chinese. A rebuild without that option keeps the saved language.
Each new report records its Browser Eval build and selected benchmark edition, with exact code and input hashes. See versioning for comparison rules and reference inputs for historical data.
To share a report online, publish its report/ directory through Artifact Site, following the hosting guide. Choose access permissions there. Formal benchmark releases also require the checks in their track guide.
Read CONTRIBUTING for code ownership, tests and PR conventions, or open an issue with a reproducible example.
make setup-test
make ciArchitecture · Security · Third-party content · Apache 2.0
Use this citation or CITATION.cff, and record the commit used.
@software{browser_eval_2026,
author = {{Browser Eval contributors}},
title = {Browser Eval},
year = {2026},
version = {0.2.0},
url = {https://github.com/lexmount/browser-eval}
}