Skip to content

fix(engine): drop hickory-dns so direct egress resolves in Docker#277

Merged
us merged 1 commit into
mainfrom
fix/engine-hickory-dns-docker
Jul 12, 2026
Merged

fix(engine): drop hickory-dns so direct egress resolves in Docker#277
us merged 1 commit into
mainfrom
fix/engine-hickory-dns-docker

Conversation

@us

@us us commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Problem

On prod (Docker) every DIRECT outbound HTTPS call from the engine failed with reqwest error sending request for url (...): managed-LLM calls (structured extract → 502), and the research APIs (OpenAlex / Semantic Scholar / arXiv / crossref) and Wikidata. Scraping was unaffected because it egresses through the HTTP proxy, which resolves the target itself.

Root cause: the workspace reqwest client is built with the hickory-dns feature, so hickory (trust-dns) is the default resolver. In the container it cannot resolve external hostnames (notably the managed-LLM endpoint, a CNAME chain) through Docker's embedded DNS at 127.0.0.11.

Isolation (all on the prod box):

  • glibc getaddrinfo resolves the host fine (getent 6/6; curl from an identical container gets HTTP 200).
  • The engine's reqwest fails every time.
  • Adding the host to the container /etc/hosts (bypassing DNS) makes extract return HTTP 200. Fault isolated to hickory DNS resolution.

Fix

Remove the hickory-dns feature from the workspace reqwest dependency (root Cargo.toml). reqwest falls back to the system getaddrinfo resolver, which reads /etc/resolv.conf natively and resolves both external CNAME chains and internal service names (searxng-internal, chrome) through Docker DNS. getaddrinfo runs on tokio's blocking pool, so it does not block the async runtime.

All 8 crates inherit reqwest = { workspace = true }, so this single change fixes every direct-egress client at once; no Rust code referenced the hickory resolver. socks/rustls and the rest of the feature set are unchanged, so the proxy scrape path is untouched. Reverts to the resolver used before the Phase B latency change; quality-neutral.

Trade-off

Gives up hickory's in-process cross-request DNS cache (the Phase B latency micro-opt). Impact is small: on prod the scrape/crawl fan-out egresses via the proxy (engine-side DNS cache never applied there), reqwest keep-alive pooling skips DNS on the hot path, and Docker/OS caches repeat lookups.

Verification

Pre-commit green locally: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo build, cargo test (1228 passed).

DNS behavior is environment-dependent (Docker embedded DNS), so it is not unit-testable in CI. Deploy-time check (with the temporary /etc/hosts pin removed):

  1. POST /api/v1/extract with a jsonSchema, paid-plan (Hobby+) key → expect HTTP 200 with structured data (was 502).
  2. GET /api/v2/search/research/papers?query=<q>&k=3 → expect HTTP 200 with non-empty hits.
  3. Sanity: a normal scrape and a search still return 200.

The workspace reqwest client used the hickory-dns resolver, which could not
resolve external hostnames (notably the managed-LLM endpoint's CNAME chain)
through Docker's embedded DNS at 127.0.0.11. Every direct outbound HTTPS call
(managed LLM, structured extract, research APIs) failed with "error sending
request"; scraping was unaffected only because it egresses through the HTTP
proxy.

Remove the hickory-dns feature so reqwest falls back to the system getaddrinfo
resolver, which reads /etc/resolv.conf natively and resolves both external
CNAME chains and internal service names through Docker DNS. getaddrinfo runs
on tokio's blocking pool, so it does not block the async runtime. Reverts to
the resolver used before the Phase B latency change; quality-neutral.
@us us merged commit 4cf098f into main Jul 12, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant