Add HTTPXWrapper by mwdd146980 · Pull Request #23822 · DataDog/integrations-core

mwdd146980 · 2026-05-22T20:10:09Z

What does this PR do?

Adds an HTTPXWrapper that implements the existing HTTPClientProtocol so checks can opt into an httpx-backed HTTP client via use_httpx: true in instance config. RequestsWrapper remains the default for every check that does not opt in.

Wires the selector into AgentCheck.http and ships a parallel unit-test suite for the wrapper. No integration opts in yet.

Motivation

This PR is part of a larger migration of datadog_checks_base's HTTP layer from requests to httpx. The goal here is design validation only: prove the wrapper-swap mechanism is architecturally sound and the new wrapper's surface works. The PR is intended to merge into a feature branch, not master, so no production users are exposed and no integration tests need to be production-faithful.

The full surface (auth tokens, proxies, UDS, Kerberos / NTLM / AWS / Digest auth, connection-pool tuning, HTTP/2, multipart uploads) and the actual per-integration migration are both deferred to follow-up PRs.

Approach

MVP feature surface in datadog_checks_base/datadog_checks/base/utils/http_httpx.py:

7 HTTP methods: get, post, put, delete, head, patch, options_method
Response surface via HTTPXResponseAdapter: .content, .text, .headers, .status_code, .json(), .raise_for_status(), .close(), .iter_lines(), .iter_content(), .encoding, .url, .cookies, .elapsed
Standard request options: headers, params, json, data, timeout, verify, cert, allow_redirects
Basic auth via username/password
TLS verify, CA bundle, client cert
Exception mapping from httpx.* into http_exceptions.* so existing widened catches keep working
Single shared httpx.Client per check instance (matches RequestsWrapper's session sharing)
Import-time ImportError if httpx is not installed (no soft-import branching)

AgentCheck.http selects between HTTPXWrapper and RequestsWrapper on is_affirmative(instance.get('use_httpx', False)). Default behavior is unchanged for every integration; the selector is wired but unused until a future PR opts something in.

Verification

Parallel unit-test suite at datadog_checks_base/tests/base/utils/http_httpx/ (51 tests, all green) covering methods, config, basic auth, TLS, exception mapping, response surface, and lifecycle.

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

Adds an httpx-backed HTTP client wrapper that implements the HTTPClientProtocol so a check can opt in via 'use_httpx: true' in instance config. RequestsWrapper remains the default. Opts kong and kuma into the new wrapper as a proof of concept. Extends mock_http_response so the existing test fixtures intercept both requests and httpx code paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

datadog-prod-us1-4 · 2026-05-22T20:11:29Z

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 11 Pipeline jobs failed

PR All | test / jcc7e1d2 / Nutanix

🔧 Fix in code (Fix with Cursor).
Undefined names 'mock_resp' and 'mocker' in tests/conftest.py, found 3 linting errors.

PR All | test / j06ca546 / SNMP

🔄 Retry job. This looks flaky and may succeed on retry.
Failed to resolve 'ddintegrations.blob.core.windows.net'.

PR All | test / j46da136 / JBoss_WildFly

🔄 Retry job. This looks flaky and may succeed on retry.
Could not resolve: ddintegrations.blob.core.windows.net:443 during startup command execution.

View all 11 failed jobs.

🧪 20 Tests failed in 1 job

PR All | run

test_bulk_table from test_check.py

(Fix with Cursor)

HTTPSConnectionPool(host=&#39;ddintegrations.blob.core.windows.net&#39;, port=443): Max retries exceeded with url: /snmp/cisco-3850.snmprec (Caused by NameResolutionError(&#34;HTTPSConnection(host=&#39;ddintegrations.blob.core.windows.net&#39;, port=443): Failed to resolve &#39;ddintegrations.blob.core.windows.net&#39; ([Errno -2] Name or service not known)&#34;))

test_cast_metrics from test_check.py

(Fix with Cursor)

HTTPSConnectionPool(host=&#39;ddintegrations.blob.core.windows.net&#39;, port=443): Max retries exceeded with url: /snmp/cisco-3850.snmprec (Caused by NameResolutionError(&#34;HTTPSConnection(host=&#39;ddintegrations.blob.core.windows.net&#39;, port=443): Failed to resolve &#39;ddintegrations.blob.core.windows.net&#39; ([Errno -2] Name or service not known)&#34;))

View all 20 test failures

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 93.16%
• Overall Coverage: 87.96% (+0.50%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 5f97af4 | Docs | Datadog PR Page | Give us feedback!}

codecov · 2026-05-22T20:31:56Z

Codecov Report

❌ Patch coverage is 93.16081% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.23%. Comparing base (dc5a6d5) to head (5f97af4).
⚠️ Report is 3 commits behind head on mwdd146980/httpx-migration-base.

Additional details and impacted files

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Fix docstring drift on mock_http_response (patches Client.request, not send). Add options['proxies'] = None for shape-parity with RequestsWrapper. Document the silent kwarg-drop policy in _build_request_kwargs. Add tests for password-only auth branch, iter_content branches, elapsed RuntimeError fallback, and AgentCheck.http httpx dispatch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Map HTTPConnectionError alongside requests ConnectionError in OM v2 scraper so ignore_connection_errors keeps working for use_httpx opt-ins - Bridge reason vs reason_phrase: prefer httpx's reason_phrase, fall back to the protocol-standard reason exposed by MockHTTPResponse - Narrow HTTPXWrapper.__del__ to AttributeError, matching RequestsWrapper - Drop unimplemented STANDARD_FIELDS keys (use_legacy_auth_encoding, tls_ignore_warning) so the config surface only lists what is honored - Strengthen response-surface tests with concrete assertions - Pin the single-shared-client invariant in test_lifecycle.py - Move use_httpx into kuma's dd_environment yield so e2e exercises the wrapper the same way kong does Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Cut stdlib-passthrough tests, parametrize duplicates, remove private-attr asserts, fold test_tls.py into test_config.py, move parse_basic_auth to common.py. 87 to 52 tests, same wrapper coverage.

… changes

…placeholder

ofek · 2026-05-29T04:57:32Z

    "cachetools==7.0.5",
    "cryptography==46.0.7",
    "ddtrace==3.19.5",
+    "httpx==0.28.1",


You'll want to use this instead https://github.com/pydantic/httpx2

yep fair. i'm going to close this in favor of #23883. it'll just be cleaner

dd-octo-sts · 2026-05-29T04:59:43Z

Validation Report

Validation	Description	Status
`dep`	Verify dependency pins are consistent and Agent-compatible	❌

Run ddev validate all changed --fix to attempt to auto-fix supported validations.

Passed validations (20)

Validation	Description	Status
`agent-reqs`	Verify check versions match the Agent requirements file	✅
`ci`	Validate CI configuration and Codecov settings	✅
`codeowners`	Validate every integration has a CODEOWNERS entry	✅
`config`	Validate default configuration files against spec.yaml	✅
`http`	Validate integrations use the HTTP wrapper correctly	✅
`imports`	Validate check imports do not use deprecated modules	✅
`integration-style`	Validate check code style conventions	✅
`jmx-metrics`	Validate JMX metrics definition files and config	✅
`labeler`	Validate PR labeler config matches integration directories	✅
`legacy-signature`	Validate no integration uses the legacy Agent check signature	✅
`license-headers`	Validate Python files have proper license headers	✅
`licenses`	Validate third-party license attribution list	✅
`metadata`	Validate metadata.csv metric definitions	✅
`models`	Validate configuration data models match spec.yaml	✅
`openmetrics`	Validate OpenMetrics integrations disable the metric limit	✅
`package`	Validate Python package metadata and naming	✅
`qa-label`	Validate the pull request declares whether it needs QA for the next Agent release	✅
`readmes`	Validate README files have required sections	✅
`saved-views`	Validate saved view JSON file structure and fields	✅
`version`	Validate version consistency between package and changelog	✅

View full run

mwdd146980 added the qa/required QA is required for this PR and will generate a QA card label May 22, 2026

dd-octo-sts Bot added integration/kong dev_package base_package integration/kuma labels May 22, 2026

mwdd146980 self-assigned this May 22, 2026

mwdd146980 and others added 7 commits May 22, 2026 17:13

phase2 mvp: add changelog fragment

72d4bc3

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

phase2 mvp: condense docstrings to teammate style

3962a15

phase2 mvp: rephrase kong/kuma httpx opt-in comments

0d65e5a

phase2 mvp: align httpx tests with teammate norms

5dc8326

Cut stdlib-passthrough tests, parametrize duplicates, remove private-attr asserts, fold test_tls.py into test_config.py, move parse_basic_auth to common.py. 87 to 52 tests, same wrapper coverage.

phase2 mvp: drop encoding-setter test (httpx passthrough)

ade5123

mwdd146980 changed the title ~~Phase 2 MVP: add HTTPXWrapper behind use_httpx flag~~ Add HTTPXWrapper behind use_httpx flag May 26, 2026

scope MVP to design validation: revert kong/kuma + mock_http_response…

f59e1b7

… changes

dd-octo-sts Bot removed integration/kong dev_package integration/kuma labels May 28, 2026

mwdd146980 changed the title ~~Add HTTPXWrapper behind use_httpx flag~~ Add HTTPXWrapper May 28, 2026

mwdd146980 added 8 commits May 28, 2026 12:22

consolidate changelog into 22676.added for feature branch merge

424f0bf

phase2 mvp: close httpx.Client explicitly in dispatch test

5d56e6d

phase2 mvp: tighten httpx wrapper kwargs + base_scraper raise

313b9de

phase2 mvp: stream httpx wrapper body via client.send(stream=True)

ac4a518

phase2 mvp: address review iteration 2 test feedback

d92d80b

phase2 mvp: drop unreachable reason fallback

8e8c4a5

phase2 mvp: drop allow_redirects per-request kwarg + comment proxies …

b6ec5ae

…placeholder

phase2 mvp: route httpx.InvalidURL through HTTPInvalidURLError

66f2302

mwdd146980 added 2 commits May 28, 2026 23:05

phase2 mvp: align HTTPXResponseAdapter with HTTPResponseProtocol shape

fd2e807

phase2 mvp: parametrize auth/dispatch tests, cover default-wrapper path

5f97af4

ofek requested changes May 29, 2026

View reviewed changes

temporal-github-worker-1 Bot added agent/review-requested ecosystems/review-requested product/review-requested labels May 29, 2026

mwdd146980 mentioned this pull request May 29, 2026

Add HTTPXWrapper backed by httpx2 #23883

Draft

3 tasks

mwdd146980 closed this May 29, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HTTPXWrapper#23822

Add HTTPXWrapper#23822
mwdd146980 wants to merge 19 commits into
mwdd146980/httpx-migration-basefrom
mwdd146980/phase2-httpx-wrapper-mvp

mwdd146980 commented May 22, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-4 Bot commented May 22, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

Uh oh!

codecov Bot commented May 22, 2026 •

edited

Loading

Uh oh!

ofek May 29, 2026

Uh oh!

mwdd146980 May 29, 2026

Uh oh!

dd-octo-sts Bot commented May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mwdd146980 commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Approach

Verification

Review checklist (to be filled by reviewers)

Uh oh!

datadog-prod-us1-4 Bot commented May 22, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

codecov Bot commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ofek May 29, 2026

Choose a reason for hiding this comment

Uh oh!

mwdd146980 May 29, 2026

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented May 29, 2026

Validation Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mwdd146980 commented May 22, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented May 22, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

codecov Bot commented May 22, 2026 •

edited

Loading