Skip to content

querier: preserve UTF-8 (dotted) label names on read when sanitization is disabled#2

Open
1linkovdim wants to merge 1 commit into
nflx-build/symcache-shardprune-bucketindexfrom
nflx-build/read-path-utf8-label-names
Open

querier: preserve UTF-8 (dotted) label names on read when sanitization is disabled#2
1linkovdim wants to merge 1 commit into
nflx-build/symcache-shardprune-bucketindexfrom
nflx-build/read-path-utf8-label-names

Conversation

@1linkovdim

Copy link
Copy Markdown
Owner

Problem

We add standard Netflix infrastructure labels (nf.app, nf.region, …) to exported profiles. With validation.disable-label-sanitization=true (our config), Pyroscope stores these dotted label names verbatim and they're matchable — {"nf.app"="foo"} works. But they never appear in Grafana typeahead / label listings.

Root cause

LabelNames (and the Series label filter) drop any label name failing legacy Prometheus validation — i.e. anything with a . — unless the request carries the per-request allow-utf8-labelnames client capability.

In the v1 read path (-architecture.storage=v1) that capability never reaches the querier, where the filter runs:

  • querier.NewGRPCHandler (pkg/querier/grpc_handler.go) wraps the QuerierService mux with only K6 middleware — not ClientCapabilitiesHttpMiddleware.
  • Nothing propagates the capability over the query-frontend → querier hop (it's only ever read from an incoming request).

So GetClientCapabilities(ctx) is always !ok in the querier, and dotted names are unconditionally filtered from listings — even though they're stored and queryable. Verified against prod: LabelValues for nf.app returns 2200 values, but LabelNames (even with a correctly-formed capability header sent to the query-frontend) omits it.

Fix

Preserve non-legacy label names on read whenever label sanitization is disabled for the tenant — the same setting that keeps them verbatim on the write path. This makes read symmetric with write and works for every client (Grafana datasource query editor, dashboard variables, Drilldown, profilecli), none of which need to send the capability. The existing per-request capability behavior is preserved unchanged.

  • New Querier.keepUTF8LabelNames(ctx): true if the request carries the capability or DisableLabelSanitization(tenant) is set.
  • Rewires both LabelNames filter sites and the Series label filter to use it.
  • Adds DisableLabelSanitization(string) bool to the querier Limits interface (validation.Overrides already implements it).

Tests

pkg/querier/querier_utf8_labels_test.go — covers: sanitization-disabled preserves dotted names without a capability; sanitization-enabled filters without a capability; capability preserves regardless; no-tenant falls back to filtering. go test ./pkg/querier/ + go vet pass.

Scope / notes

  • Affects label-name listing only; matching/storage unchanged.
  • Behavior changes only where disable-label-sanitization is already true (cluster-wide for us via overrides: {}).
  • Carried fork patch — likely worth upstreaming (arguably an upstream v1-path bug), and/or also gating the v2 readpath filter (pkg/frontend/readpath/queryfrontend/) for completeness.

🤖 Generated with Claude Code

…n is disabled

Dotted label names (e.g. nf.app) are stored and matchable when the operator
runs with validation.disable-label-sanitization=true, but they were still
filtered out of LabelNames/Series label listings (and therefore Grafana
typeahead) unless the request carried the per-request allow-utf8-labelnames
client capability.

In the v1 read path that capability never reaches the querier: it is parsed by
the query-frontend HTTP middleware, but querier.NewGRPCHandler does not
re-apply ClientCapabilitiesHttpMiddleware and the capability is not propagated
over the frontend->querier hop. So dotted label names are unconditionally
dropped from listings on v1, even though they can be queried by matcher.

Preserve them additionally whenever label sanitization is disabled for the
tenant. That is the same setting that keeps dotted names verbatim on the write
path, so honoring it on read makes read symmetric with write and works for
every client (including the Grafana datasource query editor and dashboard
variables, which never send the capability). The per-request capability
behavior is preserved unchanged.

Adds Querier.keepUTF8LabelNames() and unit tests covering the capability,
the disable-label-sanitization limit, and the no-tenant case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant