Skip to content

feat: x402 crawl gateway for training crawlers - #11

Merged
ralyodio merged 1 commit into
mainfrom
feat/crawl-gateway
Oct 21, 2026
Merged

ralyodio merged 1 commit into
mainfrom
feat/crawl-gateway

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

Wires @profullstack/x402-gateway 0.11 into src/proxy.ts, ahead of the agent passthrough.

  • GPTBot and other training crawlers: 402 with PAYMENT-REQUIRED
  • /crawl: $1/day pass offer (USDC on Base/Polygon/Ethereum) plus the CoinPay "Buy a day pass" checkout
  • Browsers, verified Googlebot, AI chat fetchers: unchanged
  • MCP endpoints (any namespace path ending /mcp) are exempt: they are the agent product
  • Open: /healthz, /api/ (Hono v1 API), robots, sitemap, manifest, sw.js
  • Proxy matcher widened from the namespace paths to every page (minus static output); the passthrough still only applies to namespace paths
  • Env read at runtime (COINPAY_X402_KEY, CRAWL_PAY_TO), already in dev2 app.env

Test: src/lib/crawl-gateway.test.ts; bun test src 12 pass; next build ok.

🤖 Generated with Claude Code

Training crawlers get a 402 with PAYMENT-REQUIRED and /crawl sells a $1 day
pass (x402 USDC or CoinPay checkout). Browsers, verified search crawlers, AI
chat fetchers and MCP clients pass through; /healthz and /api/ stay open.
The proxy matcher now covers every page so the gateway sees them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ralyodio
ralyodio merged commit 2d3bb8e into main Oct 21, 2026
1 check passed
@ralyodio
ralyodio deleted the feat/crawl-gateway branch October 21, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant