[jehrr]

Problem

A run through a metered residential proxy spent traffic and time it did not need to:

• Chromium reports a proxy's HTTP 407 as Page.goto: Timeout ... exceeded. With a wrong or exhausted credential, each engine retried and timed out again for minutes, then ended as a load failure, which points at the site instead of the proxy. The 2026-10-20 canary showed exactly this: Timeout loading https://wellfound.com/role/r/software-engineer three times through the proxy.
• Every image, video and web font on a Wellfound page went through the proxy, although every row comes from JSON embedded in the page.

Resulting behaviour

• Pre-check (all three engines). When a local proxy is configured, one request goes through it before a browser is launched: a CONNECT to the target host and HEAD /robots.txt inside the tunnel. There is no third-party IP service.
  • 407: the run logs "the gateway refused the credential (or the traffic balance is exhausted)", naming WELLFOUND_PROXY (or --proxy / --proxy-file). It exits 5 and writes <out>.latest_attempt.meta.json with status: failed and stop_reason: proxy_auth_failed.
  • 402: the same, with "proxy traffic/balance exhausted" and stop_reason: proxy_balance_exhausted.
  • The credential is never printed: the proxy is shown through proxy_pool.mask.
  • <out>.meta.json and the previous output are not touched.
  • A proxy that does not answer, or refuses the tunnel for another reason, keeps the existing behaviour: a warning, then the browser's own attempt and the retry rules.
  • Nothing changes without a proxy, or over --cdp-endpoint.
• Media blocking, only through a local proxy (never over CDP, never without a proxy):
  • Playwright: context.route aborts image/media/font by resource type. It is installed in _launch_local, so every --concurrency worker gets it too.
  • pyppeteer: CDP Fetch.enable with resource-type patterns (Image/Media/Font), on its own CDP session. page.setRequestInterception is not used because current Chromium no longer has Network.setRequestInterception.
  • Selenium: CDP Network.setBlockedURLs for image/video/font file extensions.
  • --load-media (or WELLFOUND_LOAD_MEDIA=1, shipped commented out in .env.example) turns blocking off. A hidden --block-media forces it on without a proxy, for measurements and tests.

Why media is blocked by default here

• The parser reads only JSON the server embeds in the page: __NEXT_DATA__'s Apollo cache, or the detail route's own payload. product_parser.py has no reference to an image or a font.
• Cloudflare's challenge is scripts and an iframe, and neither is blocked.
• A live loaded-vs-blocked comparison was not obtainable. One free run on 2026-10-20 (Playwright headful, no proxy, /role/r/software-engineer) was refused by Cloudflare's managed challenge (Turnstile, exit 3) before any listing was served. That is this site's documented datacentre refusal, so no blocked run was attempted.

Evidence

• Offline suite: core-only venv 574 passed; playwright venv 621; puppeteer 607; selenium 607. 0 failed in each, and each engine's new checks ran in its own venv.
• Planted-fault controls. Each one went red on the named check, and the suite was green again after restoring:
  • 407 not recognised → "a 407 from the proxy is auth_failed and fatal", "gate() returns the fatal check on a 407", "playwright_scraper: a 407 exits 5 without starting a browser".
  • Route handler never aborts → "Playwright: image/media/font aborted ...".
  • gate() removed from each of playwright/selenium/puppeteer → ": a 407 exits 5 without starting a browser".
  • Blocking install removed from each of the three engines → ": media blocking through a proxy = True (got False)".
• The same proxy_check.py was checked with real browsers on a local stub page in quora-scraper#16. With blocking on, the server never saw the img, video or font, while the text, script and stylesheet loaded, on Playwright, pyppeteer and Selenium.
• No proxy and no paid call was used.

Not changed

• The canary workflow, versions and releases.
• Selenium still cannot send proxy credentials (unchanged and documented).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8