[yummybomb]

Summary

Bench task fixes and new intents from the QA of the open tool PRs.

• doordash-pizza-near-address was graded only on the answer's format, so it passed even when the delivery address was never set. Without tools, agents called DoorDash's own search_restaurants at Times Square's coordinates, never set the address, and finished in about 9 s. With Add DoorDash store and account-checked cart tools #57's tools, agents set the address as the task asks and took about 90 s, so the tools looked 10× slower for doing the task properly. The intent is now prepare (so treatment also gets Add DoorDash store and account-checked cart tools #57's set_delivery_address action), and its prompt says to set the address with the site's picker and leave it set.
  • A page check now requires the header's address line, the line right after the "DoorDash" logo text, to be the address: an optional house number, then "Times Sq" or "Times Square", and nothing else. Checked live: a street suggestion reads "4 Times Sq"; the picker's "Times Square, 1619 Broadway" place reads "Times Square"; unset, it reads "Your Address". Places named after the square that the picker also offers ("Times Square Kitchen" in Maryland, "Times Squares Square Dance Club", "Crowne Plaza Manhattan Times Square") do not count.
  • Matching "Times Sq" anywhere on the page wouldn't work. DoorDash's search_restaurants near Times Square returns "John's of Times Square", whose store page passes such a check with no address set. So does the picker's open suggestion list ("4 Times Sq, New York, NY 10036").
• shopify-app-store-listing (new): Add shopify.com App Store search_apps and get_app tools #59's tools cover the Shopify App Store, but the only shopify.com intent asks for plan pricing on www.shopify.com, where those tools never load. This intent reads one App Store listing's rating, review count and cheapest paid plan, so Add shopify.com App Store search_apps and get_app tools #59 is measured on what it does. The pricing intent is unchanged. New answer schema: schemas/app-listing.json.
• Two comparison intents that need search. The Steam and App Store intents above each name one product the model can open from memory, so they mostly measure whether it knows the URL.
  • steam-compare-series: base-game prices and the app header's English Reviews, explicitly not Recent Reviews, for Civilization V, VI and VII (schemas/game-prices.json). The prompt and rules require the canonical labels Civilization V, Civilization VI and Civilization VII, so full store titles and trademark symbols cannot produce false disagreements between otherwise equivalent answers.
  • shopify-app-store-compare: the first three non-ad App Store results for "subscriptions", with rating, review count and cheapest paid plan (schemas/app-listings.json).
• Comparison agreement includes the complete ordered answers. Shopify compares each app's name, rating, review count and plan price; Steam compares each game's name, price and review summary. Changing metadata, swapping prices between games or changing result order now flags disagreement. Strings compare without ™ ® ℠ ©, which agents copy inconsistently from names such as "Appstle℠ Subscriptions App".
• End-state reads retry across a navigation. When an agent finishes while its last action is still loading a page, the runner now retries the end-state read on the new page (up to three tries in all, 3 s apart, in both arms). Before, the "Execution context was destroyed" error aborted the trial and reran the pair; that happened in 5 GPT-6 Luna DoorDash trials. Other read errors still abort at once.

Testing

• node --test bench/test/: 130 pass. A new run test covers the retried read and fails without the change. The stub's unreadable end state now uses an error that isn't a navigation, so the abort test still covers an immediate abort.
• Regression tests exercise both comparison intents through gradeAnswer and pairStats, covering identical answers, changed metadata, reassigned game prices and reversed results. The Steam regression also checks canonical-label enforcement, the explicit review scope, and that fixed game identities do not mislabel dynamic prices and reviews as pinned ground truth.
• node scripts/validate-registry.mjs: 265 tools valid.
• The DoorDash check was graded against page text captured live: it passes with the address set (home, search and store pages) and fails on the unset store page, the open picker and the landing page.
• All intents ran with Haiku 5.5 and GPT-6 Luna (low thinking) during the QA re-run. Comparison results, with steampowered.com: add get_app, get_search_results and search_store #134's and Add shopify.com App Store search_apps and get_app tools #59's tools (5 pairs, paired medians):
  • Steam, Haiku 5.5: right base prices 2/5 → 5/5, 0:36 → 0:12
  • Steam, GPT-6 Luna: right base prices 1/5 → 5/5, 0:19 → 0:14
  • Shopify, Haiku 5.5: 5/5 → 5/5, 0:47 → 0:14, turns 8 → 3.5
  • Shopify, GPT-6 Luna: 5/5 → 5/5, 0:19 → 0:11
• Performance trials were not rerun for the agreement changes or the canonical-label and review-scope prompt changes.
• QA run at adf8c0f with the header check and symbol-free agreement, tools from steampowered.com: add get_app, get_search_results and search_store #134 (c86efbc), Add DoorDash store and account-checked cart tools #57 and Add shopify.com App Store search_apps and get_app tools #59 pointed at locally; Shopify intents ran in stealth browsers, since the App Store answered a plain Kernel browser with 429 (5 pairs, paired medians, low thinking):

intent	model	correct C → T	time C → T	agreement differed
doordash-pizza-near-address	Haiku 5.5	5/5 → 5/5	1:02 → 0:29
doordash-pizza-near-address	GPT-6 Luna	0/5 → 5/5	n/a → 0:24
steam-compare-series	Haiku 5.5	5/5 → 5/5 (1/5 → 5/5 right)	0:29 → 0:08	4 of 5
steam-compare-series	GPT-6 Luna	5/5 → 5/5 (3/5 → 5/5 right)	0:32 → 0:17	2 of 5
shopify-app-store-listing	Haiku 5.5	5/5 → 5/5	0:22 → 0:10
shopify-app-store-listing	GPT-6 Luna	5/5 → 5/5	0:09 → 0:11
shopify-app-store-compare	Haiku 5.5	5/5 → 5/5	0:37 → 0:14	0 of 5
shopify-app-store-compare	GPT-6 Luna	4/5 → 5/5	0:20 → 0:13	1 of 4

• Every DoorDash trial graded correct ended with the header on "4 Times Sq". GPT-6 Luna's control trials replied {"places": []} after a few attempts at the picker and left the address unset.
• Every Steam and Shopify disagreement was a wrong control answer: Civilization VII's $49.99 Deluxe price, Civilization VI at $10.24 or Civilization V's Recent Reviews, and once $0 as every app's cheapest paid plan. No pair differed only in formatting.

Limitations

• Both comparison intents check the answer's format and ranges; Steam also requires the three canonical game labels. Prices and listings change too often to pin in rules. Every answer in the earlier trials passed the format check, including ones that reported Civilization VII's $49.99 Deluxe sale price instead of the $69.99 base game. agreement flags pairs whose two passing answers differ, but a plausible wrong price or review summary can still pass. The "right base prices" counts above were checked by hand.
• The Steam, Shopify and DoorDash intents only get tools once steampowered.com: add get_app, get_search_results and search_store #134, Add shopify.com App Store search_apps and get_app tools #59 and Add DoorDash store and account-checked cart tools #57 merge. Until then, their treatment arms run on main with no tools for those domains.

[yummybomb]

Pushed 5b2856e: the end-state read is now tried again (up to three times, in both arms) when an agent finishes while its last action is still loading a page. Before, the "Execution context was destroyed" error aborted the trial and reran the pair; that happened in 5 GPT-6 Luna DoorDash trials. Covered by a new run test, which fails without the change.

[yummybomb]

Pushed b896021: two comparison intents that need search. The existing Steam and App Store intents name one product the model can open from memory, so they measured whether it knows the URL.

• steam-compare-series: prices and review summaries of Civilization V, VI and VII.
• shopify-app-store-compare: the first three non-ad App Store results for "subscriptions", with rating, review count and cheapest paid plan.

Results with #134 and #59's tools (5 pairs, paired medians):

• Steam, Haiku 5.5 low: right base prices 2/5 → 5/5, 0:36 → 0:12
• Steam, GPT-6 Luna low: right base prices 1/5 → 5/5, 0:19 → 0:14
• Shopify, Haiku 5.5 low: 5/5 → 5/5, 0:47 → 0:14, turns 8 → 3.5
• Shopify, GPT-6 Luna low: 5/5 → 5/5, 0:19 → 0:11

Both intents are shape-only: every answer passed the format check, including ones that reported Civilization VII's $49.99 Deluxe sale price instead of the $69.99 base game.

[yummybomb]

Pushed adf8c0f:

• The DoorDash check also accepts the picker's "Times Square, 1619 Broadway" place, whose header line reads "Times Square". The old check failed it, as confirmed live. The whole line must now be the address, so places named after it ("Times Square Kitchen", "Times Squares Square Dance Club") still fail.
• Agreement compares strings without ™ ® ℠ ©, since Shopify results include names like "Appstle℠ Subscriptions App".

QA with both models (5 pairs, low thinking, tables in the description): DoorDash 5/5 → 5/5 on Haiku 5.5 and 0/5 → 5/5 on GPT-6 Luna, with every passing end state on "4 Times Sq". Every Steam and Shopify disagreement was a wrong control answer, not formatting.