fix(map): stop returning 504 with zero URLs on sites without a sitemap

[us]

The bug

POST /v1/map on a site without a sitemap burned the full 120s budget and came
back 504 with zero URLs. Reproduced on prod against news.ycombinator.com,
twice. Credits were refunded, so no billing harm, but the endpoint was unusable
on any sitemap-less site.

The engine log showed the shape of it — each of these is a different URL, each
paying the full renderer ladder from scratch:

HTTP 429 Too Many Requests from news.ycombinator.com/user?id=... -> retrying once via proxy
HTTP 403 received, escalating to JS renderer
antibot classifier flagged a block  renderer=chrome  signal=rate_limited
chrome nav budget hit; attempting partial snapshot
antibot classifier flagged a block  renderer=lightpanda  signal=structural_failure

Root causes

Five defects compounded into the outage:

1. No per-host block memory. saw_hard_block and use_proxy are
   request-local, so URL chore(main): release 0.0.13 #2 on a host that just 429'd us started over at direct
   HTTP: direct -> 429 -> proxy retry -> JS ladder -> chrome_proxy. That climb
   is the 10-20s/URL we measured, and it is the dominant cost.
2. The BFS discovery loop only looked concurrent. It took a semaphore permit
   and then awaited the fetch inline in the same iteration; nothing was ever
   spawned. Real concurrency was 1 and max_concurrency was dead weight.
3. **The BFS phase got a time budget only when the sitemap had already produced

   = 50 URLs.** HN has no sitemap, so it ran unbounded.

4. The route's tokio::time::timeout DROPPED the discovery future, throwing
   away every URL found so far. That is why a slow site produced a 504 with an
   empty result rather than a short list.
5. robots.txt was read only for its Sitemap: lines, and is_allowed() was
   handed a bare path, so query-keyed rules never matched. HN disallows /hide?,
   /vote? and /reply? — every one of them was fetched anyway, through a full
   Chrome ladder.

What changed

New crw-renderer/src/egress.rs — per-host memory of hosts that hard-block
our direct egress, as a moka TTL cache. Presence == latched, and TTL expiry is
the half-open re-probe, so there is no success hook that could erase it and a
false latch cannot decay into permanent paid-proxy egress.

The latch reorders egress, it never suppresses direct. A latched host starts
on the proxy, but a reserved slice of the deadline guarantees a direct rescue
even if the proxy is dead or hanging. Writes only ever happen on a genuine
direct attempt seeing a strong block signal (429 / cf-mitigated), so one
caller's broken proxy cannot demote another caller's healthy direct traffic onto
paid bandwidth. Below MIN_BUDGET_FOR_LATCH the latch is inert, which leaves the
5s scrape path byte-for-byte unchanged.

discover_urls — an overall_deadline now clamps every awaited phase
(robots, HEAD probes, sitemap tree, DNS resolution, each page), and discovery
returns the URLs it has when the budget runs out instead of dying with the
future. The outer timeout is now only a backstop.

BFS streams its fetches concurrently and exits the moment the URL cap is hit,
instead of paying for a whole level it no longer needs.

robots.txt is honoured for fetching only: a disallowed link is still
reported in the output, it is simply never requested. New is_url_allowed()
matches on path and query.

max_urls is a hard cap now; the base URL counts against it rather than
being appended after the cap was enforced.

Deliberate calls, flagged rather than buried

• Crawl-delay is not honoured. HN asks for 30s between fetches, which would
  mean map never finishes. Our per-host limiter stays the politeness mechanism.
  This is not new behaviour — /crawl already ignores it.
• Fewer URLs for sites that disallow paths we used to fetch. This is a real
  behaviour change and it is the correct one.
• Concurrency buys "up to ~2x", not more. Map follows same-origin links only,
  so every fetch contends for the same host's permits (per_host_max_concurrent
  • interactive reserve = 2). The per-host cap is deliberately NOT raised. The
    real win here is the egress memory, not the parallelism.

Verification

Against the live engine, not just the test suite:

	before	after
POST /v1/map news.ycombinator.com	504, 0 URLs, 120s	200, 5000 URLs, ~22s
same, timeout: 10	504, 0 URLs	200, partial list (~1800 URLs), 8s
docs.fastcrw.com	53 URLs	53 URLs, 1.4s
fastcrw.com	360 URLs	369-391 URLs

Old vs new binaries were built from the same tree and measured alternately on the
same machine with a cold browser, so the comparison is apples-to-apples.

Scrape-success red line: the latch is inert on the 5s scrape budget, so that
path is unchanged by construction, and a test asserts the latch metric does not
move there. On a 15-URL live sample old scored 13/15 and new 14/15 — no
regression signal. This is a sample, not the 1000-URL benchmark.

Full suite green (1200+ tests), clippy clean, fmt clean. The latch tests were
each verified to FAIL when their guard is removed, and run 10x with no flakes.

Follow-ups (not in this PR)

• run_crawl_inner has the identical dead-semaphore bug, so /crawl is serial
  too. Real, but /crawl works today and fixing it widens the blast radius.
• /crawl's robots check drops the query the same way (crawl.rs, one line).
  It would reduce the URL count /crawl returns, so it deserves its own
  benchmark verification.