Skip to content

browser-front.md

Latest commit

 

History

History
191 lines (180 loc) · 17.1 KB

browser-front.md

File metadata and controls

191 lines (180 loc) · 17.1 KB
Raw

The browser front: the policy, decision by decision

Code: s1a/browser/ (decision_model.py, action_space.py, probe_js.py, prompts.py), the decision-model layer in s1a/decision_models/ (docs/decision-models.md) and the HTTP transport in s1a/decision_models/wire.py. Tests: tests/test_browser_policy.py and tests/test_browser_recovery.py. The harness side (the DecisionPolicyModel Protocol, probe_for_policy and activate_page on the Playwright runtime, the policy path in create_browser_agent) is the decision-policy slot pinned in pyproject.toml. Measurements: docs/benchmarks.md.

Background

Every step of openJiuwen's browser subagent is one chat-model turn: the model reads a 12k-character <browser_state> and writes a tool call as free text. Measured at seconds per step, with targets the runtime sometimes has not registered. Jev is a System 1 decision model: one request holds a state and several choice questions, the answer holds one option per question with a probability distribution and a confidence, and the option can only be one of the offered indices. The reference implementation is browser-use/jev-ultrafast (MIT), whose observe-decide-act tick this policy follows.

Decisions

  1. A decision model fills the Model slot. BrowserDecisionModel(Model) answers a browser turn (the tool list holds browser_click) with exactly one browser_* tool call and forwards every other turn (summaries, typed values) to the wrapped chat model. DeepAgent, rails, checkpoints and the permission engine are untouched. The model behind it is a DecisionModel: TypeSafe Jev over HTTP with --model jev, Laya in process with --model laya; the policy is the same. action_space.py builds the Observation and the typed questions of a tick (build_observation, build_questions) and reads the answer back onto a candidate (interpret).

  2. The policy has its own in-page probe. probe_js.py waits for the page to settle and describes every visible control once. Each control is stamped data-opens1a=<id>; the stamp is the element's identity across ticks and the unique selector the runtime registers. browser_click(target_id) then passes the runtime's target validation. The runtime's own snapshot was not reused: 85 to 156 ms per call, empty <input> values, a 40-item cap, and a selector-uniqueness heuristic that drops most of Google's controls.

  3. Settle first, decide once. The probe returns after readyState, a 60 ms DOM-quiet window (cap 500 ms) and, after typing into a combobox, rendered autocomplete options (cap 900 ms, document-wide [role=option]). Settling uses timers; requestAnimationFrame never fires in a hidden tab. A last-resort resolver keeps an unresolved promise from reaching the driver timeout. This policy discards no answers. jev-ultrafast decides on every page change and discards stale answers, 8 to 11 per run in measurement.

  4. Hidden tabs are activated. A hidden tab throttles timers to about 1 Hz and does not render dropdowns. When the probe reports visibilityState == "hidden", the policy calls activate_page(url) once and probes again; a failed re-probe is refused like any other failed observation (item 8).

  5. The chat model generates typed values in the background before the field is reached. A TYPE_TEXT string is generated from the goal, the field, the page text and the history. With --prefetch on (BrowserPolicy.prefetch_values, the default) that call starts in the background for every editable field as soon as a probe shows it, keyed by URL host and path, stamp id and label. The value survives page-key changes within one document and is cancelled on navigation. Measured: value wait 0 ms with prefetch on (9 to 15 calls per run), 2.0 to 2.4 s off (3 calls). --goal-values on (BrowserPolicy.goal_value_cache) adds a text_value choice head over values extracted from the goal; off by default.

  6. The harness recognises a policy structurally. openjiuwen/harness/schema/decision_policy.py defines a runtime-checkable DecisionPolicyModel Protocol with one method, bind_runtime(runtime). create_browser_agent checks isinstance(model, DecisionPolicyModel): no temperature copy, no LLM-only context processors, anomaly-detection rail off, bind_runtime(browser_backend). The factory never imports this package.

  7. One _Run per task. Goal, history, pending action, prefetched values, tick counters and the settle budget live in a dataclass; a new goal or a finished run starts a fresh one and cancels the previous run's background tasks. The decision model is self._decision_model; Model.__init__ builds self._client as a real telemetry-bearing model client and inherited methods touch it.

  8. A failed observation is refused, not decided. The runtime reports a failed probe as ok=False or an error field. The policy checks the snapshot at the observation boundary (_require_observation) and raises RuntimeError("browser observation failed: ...") with a bounded one-line reason before the decision model or the chat answer can read an empty page. This keeps a broken browser (a missing library such as libatk-1.0.so.0, a closed page) from being answered DONE; the recovery refresh and the in-page action-settle/WAIT probes keep their own bounded handling. A decisions transport error, HTTP error, malformed 200 body or invalid distribution ends the turn with a BLOCKED summary that still holds URL, title, steps and page text. Response bodies never enter error messages or logs.

  9. Answers are validated in the decision-model layer. decide_many accepts only a choice among the offered ids whose distribution covers exactly those ids, sums to 1 within 0.02, and peaks at the choice, for every head of the request, including the heads the choice did not select. An unusable answer is re-asked once with the same request (DECISION_ATTEMPTS); a transport error is final. interpret then reads a validated Decision.

  10. WAIT is spent in-page. On a WAIT verdict the policy re-probes with a doubling settle window (500, 1000, 1500 ms) until the page_key changes or the streak budget is gone; a decisions request on an unchanged page returns the same answer and costs a full request. Two bounds hold together: MAX_CONSECUTIVE_WAITS (5 verdicts) and WAIT_SETTLE_BUDGET_MS (3000 ms per streak, kept on the run so repeated verdicts cannot re-grant it). An exhausted budget is terminal.

  11. Actions settle before the model is asked. When the probe after an action shows the same page_key, the policy waits in-page from 250 ms, doubling, up to ACTION_SETTLE_BUDGET_MS (1000 ms), counted against the same streak budget. Google Flights closes its date dialog 300 to 700 ms after "Done"; without this the model saw the old page and answered WAIT at request cost. Measured: 13 decisions and 2 waits per run became 12 and 1.

  12. Two rules the rewrite had lost are back in OPERATION_RULES: a filled search field is not a submitted search, and a visible Search or Submit control with its required fields filled is pressed at once. Without them the model answered DONE before pressing Search in 3 of 3 runs.

  13. Constants and their derivation (decision_model.py):

    constant value constraint
    PROBE_SETTLE_MS 500 equals the JS default; an un-escalated probe's timing is unchanged
    MAX_PROBE_SETTLE_MS 1500 load(3 s) + settle + 1 s last-resort must stay at least 1 s under the 30 s transport timeout
    WAIT_SETTLE_BUDGET_MS 3000 in-page wait one WAIT streak may spend before BLOCKED
    ACTION_SETTLE_START_MS, ACTION_SETTLE_BUDGET_MS 250, 1000 first and total post-action wait
    DECISIONS_TIMEOUT_S 5 Jev answers in 0.4 to 1.3 s through the proxy; a dead connection must not stall a step; one transport retry; S1A_DECISION_TIMEOUT_S overrides it for a slower System One server (configuration.md)
    DECISION_ATTEMPTS 2 one re-ask of the same request when an answer fails validation; a transport error is final
    BATCH_ACTION_TIMEOUT_MS 2000 a stamp lost to a re-render fails fast; the returned probe re-stamps

  14. Two decisions backends, separate keys. typesafe talks to api.typesafe.ai/v1/systemone with TYPESAFE_API_KEY and jev-latest; openrouter talks to /api/alpha/decisions with OPENROUTER_API_KEY and typesafe/jev-1.13 (TYPESAFE_API_URL and TYPESAFE_MODEL override the proxy only). Setting TYPESAFE_API_URL selects the proxy backend, which reads OPENROUTER_API_KEY; the TypeSafe key is never sent to OpenRouter. Measured: 294 to 418 ms per decision direct, 450 to 510 ms through the proxy.

  15. Batched actions (--batch on, BrowserPolicy.batch_actions, needs the unsafe_dev browser capability). Each step is one browser_run_code_unsafe call that performs the action and returns the next probe; one transport round trip per step. The target is the stamped selector from the last probe; a missing stamp re-probes and re-finds the element by role and label; an action that still fails returns the probe with error set, and the policy re-decides on fresh stamps. The runtime's target validation is skipped on this path.

  16. DONE answers from the whole page. On DONE the policy probes once more with all_text=True (up to 20,000 characters, viewport filter off) and asks the chat model for the answer; the terminal summary holds it under answer, next to a 2,000-character page_text excerpt for diagnosis. The harness caps the subagent's summary at 8,000 characters; the page text cannot be included in the final message.

  17. PRESS_ENTER, a keyboard submit. An editable searchbox, textbox or combobox that already holds a value gets a PRESS_ENTER candidate next to its TYPE_TEXT and CLICK ones; an empty field, a read-only one and a button do not. Unbatched it is one browser_press_key of Enter, which goes to the focused field, the one just typed into; batched it is target.press("Enter") on the field's own locator. The case comes from the Allrecipes batch: the query was typed, then the site's own search button was clicked four times without a page change.

  18. Dead targets are withheld. A control whose last two clicks left the page unchanged keeps its table row, marked click_did_nothing, and loses its CLICK candidate until a click on it moves the page again. The probe's occlusion check sees what covers a control now; a consent banner that returns after every navigation leaves the control looking clickable on each fresh probe. In the Allrecipes batch the run ended on the no-page-change guard after three such clicks.

  19. Bounded recovery (--rethink on, off by default; --rethink-attempts, default 3; --rethink-timeout, default 15 s). A stop signal -- stall_after actions without a page change, an A-B-A-B page loop, a fourth arrival at one URL, an over-long WAIT streak or a WAIT settle that moved nothing -- spends one attempt instead of ending the run. The attempt re-probes the page read-only through _raw_probe (never a navigation; a probe error or a denied read stops immediately) and, when the page_key really did not move, asks the chat model for a plan through the tool front's draft_plan; a changed page_key is delayed progress and skips the planner. The plan rides into the next observation only and never executes a click; the offered tools and candidates are filtered exactly as an ordinary turn filters them, so recovery cannot widen the tool set or add unsafe_dev. RecoveryLimits bounds the attempts and the seconds spent inside the refresh and planner calls, counted from time.perf_counter and cancelled by asyncio.wait_for, cumulatively and never reset by progress; the outer --max-steps and --timeout still cap the task. Only the detection windows (the history boundary the stall and oscillation guards read, the WAIT streak and the visit counts) reset after an attempt: history, ticks and the budget stay, so a plan gets a few actions before the same stall can fire again. A refresh or planner timeout, error, cancellation or an exhausted budget ends the run BLOCKED with no answer fetched; finish_decision_model reports it as a failure even if the summary carries text. A policy that still answers BLOCKED after one or more replans is a failure too: the partial answer is kept only as context, the summary carries the block reason and a next action, and the record is not mislabelled as a failed or exhausted recovery or a failed planner call. --model llm with --rethink on is rejected before the agent or browser is built. report() and decision_ticks.json keep the recovery events, attempts, seconds and termination. Failed recovery also includes a next action for the operator; refresh/planner errors record a short failure category without the provider response body. Failed and cancelled planner calls remain in the call count, with unknown token usage kept separate from zero cost. To compare, run the same goal twice, --rethink off against on, on the same backend; the on/off difference is only worth quoting once the browser bench is rerun, because the scripted tests pin the loop, not a task success rate.

  20. The final page is saved. However a task ends, a timeout included, run_task in browse.py takes one viewport PNG of the live page before it releases the browser, saves it as final.png in --logs-dir and puts that path in the answer's screenshot. A judge that grades the end state can read the file, as Harbor's WebVoyager judge does at /logs/agent/final.png. The runtime is the one create_browser_agent gave the agent's BrowserRuntimeRail, and the call is browser_take_screenshot. @playwright/mcp writes the PNG under its output directory (.playwright-mcp/ in its cwd, or --output-dir) and links it in its report, relative to its cwd. The runtime's MCP client drops the inline image, so the file is copied from that link. The link's path is written raw, so SCREENSHOT_LINK reads the whole - [Screenshot of ...](...) line and a directory such as shots (1) keeps its parentheses. A final.png an earlier run left in a reused --logs-dir is removed when the task starts, so a failed or skipped screenshot never leaves a judge the earlier page. Nothing is taken when the runtime never observed a page, because the tool would launch a new browser to take one. The call is capped at SCREENSHOT_TIMEOUT_S (5 s, @playwright/mcp's own screenshot timeout). A failure leaves screenshot null and records only the exception type in screenshot_error, because the exception text can quote the page. elapsed_ms stops before the screenshot, and neither the outcome nor the release of the browser and the session depends on it. Tests: tests/test_browse.py, and tests/system/test_final_screenshot.py on a live @playwright/mcp.

Rejected

  • A browser_jev_run tool: the chat model would still decide when to delegate, one extra turn per step.
  • Jev as a browser driver backend: a driver has eyes and hands only; the decision would be lost.
  • Reusing the runtime's snapshot probe: decision 2.
  • requestAnimationFrame for settling: never fires in a hidden tab.
  • An "Open " prefix on the click head for editable fields, as jev-ultrafast does: tried, 5 of 6 runs still chose TYPE_TEXT on the date box; withdrawn.
  • Re-asking the model on an unchanged page after a WAIT: decision 10.
  • browser_run_code_unsafe with page.screenshot({path}) for the final page, the way s1a/tool/hands.py takes frames: @playwright/mcp waits 500 ms after the code, and longer when the page made requests meanwhile.
  • browser_take_screenshot with filename set to the final page's path: @playwright/mcp refuses a file outside its cwd and its output directory, and --logs-dir can be anywhere (Harbor's is /logs/agent).
  • The process-wide _ACTIVE_BROWSER_RUNTIMES set as the final page's runtime: s1a-mcp runs one task after another in one process, and the set can still hold an earlier task's runtime.

Open issues

  1. The date box: the model picks TYPE_TEXT on a field labelled "Departure" in 4 of 9 runs and the value model then types a city. Next step: put the input's type, placeholder and autocomplete into the element table so the operation question can see "this is a date field".
  2. The declarative subagent path (SubAgentConfig) has no decision-policy option; only create_browser_agent(model) takes a policy.
  3. The decisions client is a plain httpx.AsyncClient, outside the harness's connection pool.
  4. Budget exhaustion is treated as terminal on the assumption that the model answers the same WAIT for an unchanged snapshot; if a live run shows otherwise, re-ask once before BLOCKED.
  5. The runtime's navigate, evaluate and press_key are private; s1a/tool/hands.py reaches them through _call_playwright_tool and _execute_probe_json (upstream ask in docs/roadmap.md). browse.py takes the final screenshot through _call_playwright_tool too and reads the runtime off BrowserRuntimeRail._runtime; the agent has no public handle to it.