fix(engine): drop hickory-dns so direct egress resolves in Docker#277
Merged
Conversation
The workspace reqwest client used the hickory-dns resolver, which could not resolve external hostnames (notably the managed-LLM endpoint's CNAME chain) through Docker's embedded DNS at 127.0.0.11. Every direct outbound HTTPS call (managed LLM, structured extract, research APIs) failed with "error sending request"; scraping was unaffected only because it egresses through the HTTP proxy. Remove the hickory-dns feature so reqwest falls back to the system getaddrinfo resolver, which reads /etc/resolv.conf natively and resolves both external CNAME chains and internal service names through Docker DNS. getaddrinfo runs on tokio's blocking pool, so it does not block the async runtime. Reverts to the resolver used before the Phase B latency change; quality-neutral.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On prod (Docker) every DIRECT outbound HTTPS call from the engine failed with reqwest
error sending request for url (...): managed-LLM calls (structured extract → 502), and the research APIs (OpenAlex / Semantic Scholar / arXiv / crossref) and Wikidata. Scraping was unaffected because it egresses through the HTTP proxy, which resolves the target itself.Root cause: the workspace
reqwestclient is built with thehickory-dnsfeature, so hickory (trust-dns) is the default resolver. In the container it cannot resolve external hostnames (notably the managed-LLM endpoint, a CNAME chain) through Docker's embedded DNS at127.0.0.11.Isolation (all on the prod box):
getaddrinforesolves the host fine (getent6/6;curlfrom an identical container gets HTTP 200)./etc/hosts(bypassing DNS) makes extract return HTTP 200. Fault isolated to hickory DNS resolution.Fix
Remove the
hickory-dnsfeature from the workspacereqwestdependency (rootCargo.toml). reqwest falls back to the systemgetaddrinforesolver, which reads/etc/resolv.confnatively and resolves both external CNAME chains and internal service names (searxng-internal, chrome) through Docker DNS.getaddrinforuns on tokio's blocking pool, so it does not block the async runtime.All 8 crates inherit
reqwest = { workspace = true }, so this single change fixes every direct-egress client at once; no Rust code referenced the hickory resolver.socks/rustlsand the rest of the feature set are unchanged, so the proxy scrape path is untouched. Reverts to the resolver used before the Phase B latency change; quality-neutral.Trade-off
Gives up hickory's in-process cross-request DNS cache (the Phase B latency micro-opt). Impact is small: on prod the scrape/crawl fan-out egresses via the proxy (engine-side DNS cache never applied there), reqwest keep-alive pooling skips DNS on the hot path, and Docker/OS caches repeat lookups.
Verification
Pre-commit green locally:
cargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings,cargo build,cargo test(1228 passed).DNS behavior is environment-dependent (Docker embedded DNS), so it is not unit-testable in CI. Deploy-time check (with the temporary
/etc/hostspin removed):POST /api/v1/extractwith ajsonSchema, paid-plan (Hobby+) key → expect HTTP 200 with structured data (was 502).GET /api/v2/search/research/papers?query=<q>&k=3→ expect HTTP 200 with non-empty hits.