[Feat] Confidence-gated deferral: hand low-confidence browser steps to the chat model and measure the trade-off #94

Why

The README's pitch is that a System 1 decision model answers each step with probabilities, so "calling code can act above a threshold and escalate below it". In the browser front today nothing does that: BrowserDecisionModel (s1a/browser/decision_model.py) always acts on the decision model's top choice, and the wrapped chat model (fallback) is only used for non-browser turns, typed values and the bounded-recovery plan. The per-step confidence is recorded but never used.

Recent work shows the gate is where most of the cost/quality trade-off sits:

• ReDAct: Uncertainty-Aware Deferral for LLM Agents (arXiv:2604.07036): a small model acts by default and defers a step to a large model when its uncertainty exceeds a threshold calibrated on 100 episodes for a target number of large-model calls per episode. On ALFWorld, deferring about 15% of decisions matches using the large model throughout.
• Think Short, Defer Smart (arXiv:2607.26865): calibrates the deferral rule on whole-episode trajectories (Learn-Then-Test) to get finite-sample guarantees on both reward and cloud-call rate.
• SearchJev (arXiv:2610.05107), already cited in [Decisions] Make confidence mean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin) #89, does confidence-gated System-1 to System-2 delegation with Jev.

#89 lists exactly this as its follow-up ("Adding a confidence gate or System-2 fallback ... once the number means one thing"), and #91 makes confidence = max_k p_k on every backend, so the number is now comparable across Jev, Laya, CLM and Cua.

Proposed scope (one PR, off by default)

1. A browser-front policy setting, e.g. --defer-below P / S1A_DEFER_BELOW (unset = today's behaviour). When the settled step's confidence is below P, the step is decided by the chat model already wired as fallback, given the same candidates the decision model saw, and must return one of them (same validation as today).
2. Step records gain deferred: true|false, the decision model's choice and confidence, and the chat model's choice, so every deferral can be audited and the "would it have been right" question answered from the trace.
3. An offline calibration helper over saved runs: for a target deferral rate (e.g. 10%, 15%, 25% of steps), report the threshold that gives it on that backend. No live calls.
4. One live comparison on an existing task (e.g. the WebVoyager Allrecipes task in the README, or the deterministic local search scenario from [Tests] Add deterministic local browser search scenario #63): decision model only, chat model only, and deferral at one calibrated rate; success, wall time, cost and the deferral rate, with the runs committed under docs/results/.

Acceptance criteria

• With the setting unset, behaviour and records are unchanged (existing tests pass as-is).
• Offline tests with fakes: a low-confidence step goes to the fallback, a high-confidence one does not, a fallback answer outside the candidates is rejected the same way a bad decision is.
• Thresholds are documented as backend-specific and not calibrated probabilities (per [Decisions] Make confidence mean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin) #89).
• Results table shows the cost/speed/score trade-off honestly, including if deferral does not help.

Out of scope

• Deferral in the desktop, games and robotics fronts (follow-ups once the browser result is in).
• Learned or conformal calibration (TSDS-style guarantees); a plain empirical threshold is enough for a first measurement.

References

• ReDAct: https://arxiv.org/abs/2604.07036
• Think Short, Defer Smart: https://arxiv.org/abs/2607.26865
• SearchJev: https://arxiv.org/abs/2610.05107
• Related: [Decisions] Make confidence mean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin) #89 / [Fix] Unify decision confidence as max_k p_k (keep backend-native as backend_confidence) #91 (confidence semantics), [Feat] Recover from stalled browser and desktop tasks with bounded replanning #16 / [Feat] Add bounded recovery for stalled browser and desktop tasks #31 (bounded recovery uses the chat model as planner), [Tests] Add deterministic local browser search scenario #63 (local search scenario)