[jehrr]

Problem

A residential proxy is billed by the byte, and a run had no idea how many bytes it had used. A slow page, a retry loop or a long paginated run could spend a proxy balance with no record anywhere except the provider's invoice. Nothing capped it either.

What changes

Only when traffic goes through a local proxy (--proxy / --proxy-file / WELLFOUND_PROXY). Nothing changes without a proxy or over --cdp-endpoint (the 2Captcha Scraping Browser API has its own billing).

• Meter (proxy_meter.py), in all three engines:
  • Playwright: the sum of request.sizes() (request headers + body, response headers + body) on requestfinished / requestfailed.
  • pyppeteer: CDP Network.loadingFinished.encodedDataLength (bytes received, headers included) plus each request's line, headers and body from Network.requestWillBeSent, on a CDP session of its own. Requests the browser blocked itself (media blocking) cost nothing.
  • Selenium: the same CDP events, read from chromedriver's performance log (now requested at driver start) after each page and before the driver quits.
  • It is an estimate of the payload through the tunnel. TLS, the CONNECT exchange and the browser's own background connections are not visible to any engine and add a few percent. The pre-check (under 1 KB) is not counted.
• Budget: --proxy-budget-mb (default 10, 0 = no cap, WELLFOUND_PROXY_BUDGET_MB). Negative or non-numeric values are a usage error (exit 2). An unparseable env value stops the run rather than falling back. Once the total is over the budget:
  • Playwright aborts every new request in a route registered after the media route, so it runs first.
  • pyppeteer and Selenium send Network.setBlockedURLs ["*"].
  • Every engine raises at its next navigation or retry.
  • Playwright's --concurrency workers share one meter (its add now takes a lock). The first worker to hit the cap stops dispatch for all of them.
  • Rows already gathered are kept, stop_reason: proxy_budget_exceeded, exit 6 with rows and exit 5 without.
  • The stop is a BaseException, so the engines' broad except Exception retry loops cannot swallow it and retry.
• Sidecar: proxy_bytes, proxy_budget_bytes (null with no cap) and proxy_budget_exceeded go in <out>.meta.json. Every metered run also writes <out>.latest_attempt.meta.json, so a run that wrote no rows still records what it spent. Without a proxy no such file is written, as before. The log ends with Proxy traffic: X.X MB of Y MB budget.
• Canary: both jobs pass --proxy-budget-mb 10: listing, through WELLFOUND_PROXY, and the bare datacentre job, which has no proxy and reports "not metered". After each run, .github/proxy_traffic.py appends **listing**: proxy traffic X.X MB of 10 MB to $GITHUB_STEP_SUMMARY. When the cap was hit it prints ::error::stopped: proxy budget 10 MB exceeded and fails.
• Env precedence fix: env_config.apply filled any falsy destination, so an explicit --proxy-budget-mb 0 would have been replaced by WELLFOUND_PROXY_BUDGET_MB. A typed number is now kept. Every mapped default is None, False or "", so a number can only have come from the command line. The variable ships commented out in .env.example.
• pyproject.toml lists proxy_check (missing since the savers PR) and proxy_meter.
• README (proxy savers section, output contract), --help, .env.example, Dockerfile COPY list, CHANGELOG [Unreleased].

Evidence

• Offline suite, core-only venv: python smoke_test.py → 0 failed. The new groups cover:

  • the meter against known Playwright sizes() and CDP events (exact byte counts), plus blocked, failed and redirect cases;
  • an HTTP client through a byte-counting stub proxy: 0.1% drift;
  • the sidecars and exit codes 6/5;
  • the canary steps, plus the summary script run against over-budget, under-budget, no-proxy and missing sidecars.

• Engine venvs (playwright 1.64, pyppeteer, selenium; no browser download), with the opt-in real-browser check (PROXY_METER_BROWSER=1, local Chromium): each suite → 0 failed (679 / 664 / 664 checks passed). Each engine:

  • installs the meter through a proxy and not without one;
  • refuses to navigate over budget;
  • driven past the budget on page 1 of 3, exits 6 with the rows kept and no third page; with no rows, exits 5 with the attempt sidecar;
  • Playwright, with --concurrency 2 past the budget on page 1: exit 6, and the workers stop at their first navigation.

• Real browser, through a local stub proxy that counted every byte it carried (60 kB page + 30 kB CSS + 90 kB JS):

  Driver	Metered	Stub proxy carried	Drift
  Playwright	181,007 B	181,433 B	0.2%
  pyppeteer	180,859 B	181,936 B	0.6%
  Selenium	181,167 B	182,431 B	0.7%

  Past a 0.15 MB budget, a request the page made never reached the proxy, and check() stopped the run.

• Planted-fault controls. Each went red on the named check and was green again after restoring:

  • meter disabled (ProxyMeter.add counts nothing): "1 MB budget: exactly 1,000,000 bytes…" and the following checks;
  • budget check disabled (proxy_meter.check no-op): "proxy_meter.check(args) stops a run that is over budget";
  • the check before navigating removed, in each engine and run in that engine's venv: ": over budget, the page fetch stops before navigating";
  • the listing run's reporter removed from canary.yml: "each run's proxy traffic goes to the step summary…";
  • the explicit-0 precedence fix reverted: "an explicit --proxy-budget-mb 0 wins over the environment";
  • the summary script no longer failing over budget: "over budget: '::error::stopped: proxy budget 10 MB exceeded'…".

• No proxy, no paid call and no canary dispatch was used.

Not changed

• Nothing for runs without a proxy or over --cdp-endpoint.
• No release or version bump.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8