Skip to content

fix(http): prevent duplicate transport-owned headers#3567

Draft
djwhitt wants to merge 4 commits into
mainfrom
davidwhittington/o11y-1961-webhook-log-drains-duplicate-content-type-header-breaks-body
Draft

fix(http): prevent duplicate transport-owned headers#3567
djwhitt wants to merge 4 commits into
mainfrom
davidwhittington/o11y-1961-webhook-log-drains-duplicate-content-type-header-breaks-body

Conversation

@djwhitt

@djwhitt djwhitt commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Problem

When an HTTP backend is configured with a user-supplied Content-Type header, requests went out with two Content-Type headers (e.g. application/jsonapplication/json). Tesla middleware set the header by appending, and Tesla.put_headers/2 appends rather than de-duplicates, so the user's header survived alongside the transport's. Receivers concatenated the two values, failed to parse the body, and delivered an empty body.

Two paths exhibited this:

  • Webhook-derived backends (Datadog, Loki, Elastic, Incident.io, …): Tesla.Middleware.JSON appends content-type when it encodes the body.
  • OTLP backends: OtlpAdaptor passes user-configured headers alongside ProtobufFormatter, which always sets content-type to application/x-protobuf. A user-supplied content-type survived next to the formatter's.

Fix

Give transport-owned headers a single source rather than de-duplicating by header order (which is non-deterministic).

Shared plumbing:

  • New Logflare.Backends.Adaptor.HttpBased.Headers with drop_reserved/2 (case-insensitive strip) and normalize_keys/1.

Webhook path:

  • WebhookAdaptor.Client.send/1 strips the reserved set per request: content-type only when the JSON middleware will actually encode the body — binary payloads (e.g. NDJSON) keep a custom content-type — and content-encoding when gzip is enabled. authorization is intentionally not reserved here, since several adaptors set it directly.
  • WebhookAdaptor.cast_config/2 lowercases header keys so stored config cannot hold case-variant duplicates of the same header.
  • The redacted-secret restore path looks up existing values by normalized key, so a stored secret is not dropped when its casing differs from the submitted (possibly normalized) key.

HTTP-based / OTLP path:

  • HttpBased.Client.new/1 strips content-encoding (gzip) and authorization (token/basic auth).
  • The body is not known at client-build time, so the client cannot infer whether the JSON middleware will own content-type. Instead, a formatter that sets content-type itself declares ownership via an optional reserved_headers/0, and the client drops those names from user headers regardless of body. OtlpAdaptor.ProtobufFormatter and SentryAdaptor.EnvelopeBuilder declare ["content-type"].
  • OtlpAdaptor.cast_config/2 lowercases header keys, matching the webhook path.

When a user supplies a Content-Type header on a webhook/log-drain backend,
Tesla's JSON middleware appended a second one and Tesla.put_headers/2 does not
dedupe, so requests went out with two Content-Type values. Receivers
concatenated them into an unparseable value and delivered an empty body.

Give transport-owned headers a single source instead of relying on header
ordering:

- Add Logflare.Backends.Adaptor.HttpBased.Headers with drop_reserved/2 and
  normalize_keys/1.
- WebhookAdaptor.Client.send/1 strips the reserved set per request: content-type
  only when the JSON middleware will encode the body (binary bodies keep a custom
  content-type), content-encoding when gzip is enabled.
- HttpBased.Client.new/1 strips content-encoding (gzip) and authorization
  (token/basic auth).
- WebhookAdaptor.cast_config/2 lowercases header keys so stored config cannot
  hold case-variant duplicates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
djwhitt and others added 3 commits June 8, 2026 19:10
HttpBased.Client deliberately excluded content-type from its reserved
header set because the body is not known at client-build time, so it
could not tell whether Tesla.Middleware.JSON would encode and own it.
But OtlpAdaptor passes user-configured headers alongside ProtobufFormatter,
which always sets content-type to application/x-protobuf. A user-supplied
content-type therefore survived next to the formatter's, reintroducing the
duplicate-header failure mode for OTLP backends.

Let the formatter that owns the header declare it instead of having the
client guess what middleware will do:

- HttpBased.Client.reserved_header_names/1 folds in an optional
  reserved_headers/0 exported by the formatter module (handles both the
  module and {module, opts} forms).
- OtlpAdaptor.ProtobufFormatter and SentryAdaptor.EnvelopeBuilder declare
  reserved_headers/0 => ["content-type"].
- OtlpAdaptor.cast_config/2 lowercases header keys (parity with
  WebhookAdaptor) so stored config cannot hold case-variant duplicates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
unredact_headers/2 looked up the stored secret by an exact key match. When
stored config predates normalize_keys/1 (e.g. "Authorization") and the
submitted redacted key uses different casing (e.g. "authorization"), the
lookup missed and the secret was silently dropped before normalization ran.

Normalize the existing headers and look up by downcased key so the restore
path is case-insensitive, matching the rest of the header handling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@djwhitt djwhitt changed the title fix(webhook): prevent duplicate transport-owned headers in HTTP backends fix(http): prevent duplicate transport-owned headers Jun 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant