feat(core): add page exploration tools, structured extract

[lmorchard]

Summary

• Adds two zero-LLM page-inspection tools (search_page, find_elements) so the agent doesn't pay an LLM round-trip for inventory questions ("is X on the page?", "how many Y are there?").
• Extends extract with an optional outputSchema (JSON Schema) so the agent can request structured JSON output instead of markdown text it then has to re-parse.
• Implemented across both browser backends — PlaywrightBrowser (CLI/server, with iframe iteration) and ExtensionBrowser (top-frame only, matching the extension's existing getTreeWithRefs scope).

Design Decisions

• Dedicated AriaBrowser methods rather than routing through PageAction + performAction. Typed input/output, no string-encoded options. Mirrors extract's existing bypass of performAction.
• New inspectionTools.ts factory alongside webActionTools/searchTools/tabstackTools/interactiveToolSet. Matches the per-concern file pattern; signals read-only intent.
• Refs via the existing data-pilo-ref DOM attribute. The aria-tree code already sets this attribute during tree generation; new tools resolve nearestRef via el.closest('[data-pilo-ref]') and withinRef lookups via document.querySelector('[data-pilo-ref="..."]'). No changes to the aria-tree bundle.
• Frame iteration matches each backend's existing aria-tree behavior. Playwright iterates same-origin + accessible cross-origin frames and tags per-frame matches with frameUrl; Extension is top-frame only (so frameUrl is always undefined in extension results). Forcing parity in either direction would require either dropping Playwright's frame coverage or adding allFrames: true machinery to the extension.
• Wiring is unconditional. These tools have no API key / callback / provider dependency, so they're always in the agent's tool set — unlike searchTools (gated on search service), tabstackTools (gated on API key), and interactiveToolSet (gated on callback).
• Structured extract routes through generateObjectWithRetry with the AI SDK's jsonSchema() helper. Markdown branch (no outputSchema) is byte-identical to its pre-PR behavior. Complementary to tabstack_extract_json — that one is for off-page URL fetches via the Tabstack API; this is for the current page using the configured LLM provider.

Changes

Core (packages/core/src/):

• tools/inspectionTools.ts (new) — createInspectionTools factory exposing search_page and find_elements.
• browser/ariaBrowser.ts — new types (SearchPageOptions/Match/Result, FindElementsOptions/Match/Result) and two new interface methods (searchPage, findElements).
• browser/playwrightBrowser.ts — searchPage and findElements implementations with cross-frame iteration mirroring getTreeWithRefsImpl.
• tools/webActionTools.ts — extract extended with optional outputSchema.
• utils/retry.ts — new generateObjectWithRetry + shared retryDriver<T> helper extracted from both wrappers. NoObjectGeneratedError is now non-retryable.
• webAgent.ts — createInspectionTools instantiated unconditionally; search_page and find_elements added to the pageChanged exempt list.
• prompts.ts — three new TOOL_STRINGS entries (searchPage, findElements, extended extract); three buildToolExamples lines; one best-practices bullet in actionLoopSystemPromptTemplate.
• core.ts — new type exports.

Extension (packages/extension/src/):

• background/ExtensionBrowser.ts — searchPage and findElements implementations via browser.scripting.executeScript, top frame only.

Tests: +1305 across packages (core 719, cli 221, server 88, extension 277). New inspectionTools.test.ts (446 lines) and new describe blocks in playwrightBrowser.test.ts, ExtensionBrowser.test.ts, retry.test.ts, webActionTools.test.ts. Includes NoObjectGeneratedError non-retry coverage.

Test Plan

• pnpm run check passes (typecheck + format:check + 1305 tests across all packages)
• gitleaks detect — no leaks
• Manual smoke: run pnpm pilo run "<task>" against a static page (e.g., a Wikipedia article) and a dynamic page (e.g., a documentation search results page) and confirm:
  • search_page returns matches with nearestRef populated
  • find_elements with a CSS selector returns elements with auto-resolved href/src
  • find_elements with withinRef scopes correctly
  • extract without outputSchema still returns markdown
  • extract with outputSchema returns a JSON object matching the schema

References

• Spec: docs/dev-sessions/2026-05-13-1319-page-exploration-tools/spec.md (not committed to git — session artifact)
• Plan: docs/dev-sessions/2026-05-13-1319-page-exploration-tools/plan.md (not committed to git — session artifact)
• Closes Add zero-LLM page exploration tools (search_page, find_elements, structured extract) #432

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds two zero-LLM page-inspection tools (search_page, find_elements) and extends extract with optional outputSchema for structured JSON output, implemented across both Playwright and Extension browser backends.

Changes:

• New search_page and find_elements tools (via new inspectionTools.ts factory and new AriaBrowser methods), with cross-frame iteration in Playwright and top-frame-only in the Extension.
• extract extended with optional outputSchema routed through a new generateObjectWithRetry helper, with NoObjectGeneratedError treated as non-retryable.
• retry.ts refactored to share a retryDriver<T> between text and object wrappers; prompts updated with new tool examples and best-practices guidance.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
packages/core/src/browser/ariaBrowser.ts	New types and interface methods for searchPage/findElements.
packages/core/src/browser/playwrightBrowser.ts	Playwright implementations with main + sub-frame iteration.
packages/core/src/tools/inspectionTools.ts	New factory exposing the two zero-LLM tools.
packages/core/src/tools/webActionTools.ts	extract extended with structured-output branch.
packages/core/src/utils/retry.ts	Shared retry driver + new generateObjectWithRetry; NoObjectGeneratedError non-retryable.
packages/core/src/webAgent.ts	Wires inspection tools unconditionally; adds them to pageChanged exempt list.
packages/core/src/prompts.ts	Tool descriptions, examples, best-practices line.
packages/core/src/core.ts	Exports new types.
packages/extension/src/background/ExtensionBrowser.ts	Extension implementations via scripting.executeScript.
packages/core/test/**, packages/extension/test/ExtensionBrowser.test.ts	New unit tests covering both backends, retry, and structured extract.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.