Skip to content

feat(symbolizer): expose address-level symbol resolution as a public API#5317

Open
jake-kramer wants to merge 7 commits into
mainfrom
symbolizer/resolve-seam
Open

feat(symbolizer): expose address-level symbol resolution as a public API#5317
jake-kramer wants to merge 7 commits into
mainfrom
symbolizer/resolve-seam

Conversation

@jake-kramer

@jake-kramer jake-kramer commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Part of a series: symbol-aware tree references (1 of 8)

The full picture. Native and eBPF profiles are ingested with raw memory addresses; function
names are resolved at query time from debug info. Today that resolution only triggers when the
query-frontend sees an __unsymbolized__ dataset label — which queryless queries ({}) never
surface, so their results silently lose symbolization — and it works by rewriting the whole TREE
query into PPROF and back, which makes every dataset in the query pay the pprof cost and silently
drops any query selector that isn't hand-mirrored between the two query types. This series changes
the tree representation instead: tree nodes can reference unresolved {buildID, address} locations
directly, partial trees carry those references through every merge (with truncation deferred), and
the frontend resolves them once after the final merge, then rewrites and truncates the tree.
Correct for every query shape, no pprof detour, and the wire format stays a tree. Tracking issue:
grafana/pyroscope-squad#487.

The series, in order (each PR stands alone for review; no new behavior activates until 6):

  1. symbolizer: expose address-level Resolve API ← this PR.
    Extracts per-address symbol lookup from behind SymbolizePprof into a public Resolver
    interface that later PRs consume directly. Pure refactor: no caller change, no new behavior.
  2. proto: SymbolRefTable + tree query/report fields.
    The wire schema for trees that carry unresolved references — a compact side table of resolved
    names, interned build IDs, and addresses. Schema + generated code only; no consumers yet.
  3. model/symbolref: intern table + merge rebasing.
    New package implementing the reference table and deterministic reference rebasing when partial
    trees from different backends merge.
  4. model/symbolref: job grouping + rebuild.
    Groups unresolved references into per-build-ID resolution jobs, and rebuilds the final resolved
    tree — expanding inline-frame chains, merging newly-identical nodes, truncating exactly once.
  5. symdb + querybackend: produce and aggregate symbol-ref trees.
    Backends emit trees with unresolved references for unsymbolized datasets (deferring truncation)
    and merge partials of both shapes; fully-symbolized datasets keep today's exact path.
  6. frontend: orchestration behind a default-off per-tenant flag.
    The frontend requests symbol-ref trees, resolves post-merge via this PR's API, and rewrites the
    result. The first PR where any new behavior can be switched on.
  7. integration tests + docs. (test(symbolizer): add symbol-ref tree integration tests and docs #5335)
    End-to-end coverage — including the queryless-query case that motivated the issue — plus an
    equivalence test pinning the new path's output against the legacy one.
  8. symdb: compaction preserves line-less locations (fix(symdb): preserve line-less locations when rewriting partitions without functions #5337) — merges first.
    Fixes a pre-existing bug (fix: drop malformed locations #4051) in the shared compaction symbols rewriter that replaced every
    location of a partition without functions — exactly the fully-unsymbolized native datasets this
    series serves — with a zero value, permanently destroying the addresses symbolization needs.
    Found during end-to-end validation of this series; stacked at the bottom so it lands before the
    feature can be enabled anywhere.

What / Why

pkg/symbolizer could previously only be driven through SymbolizePprof, which takes a whole
pprof profile. Anything that wants symbols for a set of {buildID, address} pairs — for example,
resolving symbols for a tree-shaped query result — had no way to ask for that directly; it would
have had to synthesize a throwaway pprof profile just to reuse the lookup path.

This PR extracts that lookup into a narrow, exported interface:

type Resolver interface {
    Resolve(ctx context.Context, buildID, binaryName string, addrs []uint64) ([][]lidia.SourceInfoFrame, error)
}

SymbolizePprof is refactored to call Resolve internally instead of duplicating the lookup
logic. Resolve is defined on *Symbolizer in the producer package (pkg/symbolizer), the same
pattern pkg/objstore.Bucket already uses, so a later out-of-process implementation can satisfy
Resolver without importing this package.

What moved: fallback-symbol synthesis (the binary!0xaddr placeholder text). Resolve itself
never produces it — a nil/empty slot in its result means "could not resolve this address," full
stop. Rendering the placeholder is now the caller's job: SymbolizePprof's per-mapping goroutine
calls the existing createFallbackSymbol helper immediately after Resolve returns, for every nil
slot. This is a mechanical relocation of an existing call, not a change to what it computes.

What stayed byte-identical:

  • SymbolizePprof's emitted profile, on every existing fixture — same metric names/label values,
    same log message text, same check order as the code this replaces.
  • The pre-existing, deliberately-unreused lookup buffer in the per-address loop
    (var framesBuf []lidia.SourceInfoFrame is never reassigned, so each table.Lookup call gets
    nil) — preserved exactly as-is; not something this PR changes or "fixes."

Behavior changes

There is exactly one externally-visible behavior change:

Context cancellation or a deadline firing while Resolve is resolving symbols now surfaces as a
real error, and that error now propagates out of SymbolizePprof instead of being silently
swallowed into a fallback-filled profile with a nil error.

Concretely: Resolve returns a non-nil error only when ctx.Err() != nil (checked at entry
and again around the debug-info fetch, via errors.Is against context.Canceled /
context.DeadlineExceeded); every other failure mode (empty/invalid buildID, fetch failure,
malformed debug info, a specific address not resolving) keeps degrading to a nil/empty result slot
with a nil error, exactly like the code it replaces.

This matters today at the three real call sites that invoke SymbolizePprof, all of which already
treat any error it returns as fatal to the request:

  • pkg/frontend/readpath/queryfrontend/symbolizer.go:82
  • pkg/frontend/readpath/queryfrontend/query_select_merge_profile.go:112
  • pkg/frontend/readpath/queryfrontend/query_select_merge_profile.go:237

Before this PR, a query whose context was canceled or timed out mid-symbolization would silently
get back a profile full of binary!0xaddr placeholders and no error. After this PR, that same
request now fails with a wrapped context.Canceled/context.DeadlineExceeded error, matching how
these call sites already handle every other symbolization failure.

Internal contract note (not observable from outside pkg/symbolizer, since SymbolizePprof's
output is unaffected): whenever Resolve returns a nil error, the returned outer slice always has
len(result) == len(addrs), and result[i] == nil means address i didn't resolve, for any reason
short of context cancellation.


Security hardening

Resolve is a new public entry point into a buildID-driven object-store path
(getLidiaBytesfetchLidiaFromObjectStorepath.Join(bucketPrefix, tenantID, buildID)).
Previously, this path was only reachable through SymbolizePprof, whose one caller always
sanitized buildID first — so encapsulation, not validation, was what kept it safe. Now that
Resolve is exported, Resolve validates buildID itself (same sanitizer extractBuildID already
applies) before doing any object-store or debuginfod I/O, and degrades to a nil-per-miss result
instead of trusting the caller. A test asserts zero mock calls on the storage/debuginfod clients
when buildID fails validation, proving the rejection happens before any I/O — the property that
actually matters for a future caller that might source buildID from less-trusted input.


Testing

  • Full pkg/symbolizer suite passes under -race, including the pre-existing golden-baseline
    tests (TestSymbolizePprof, TestSymbolizationKeepsSequentialFunctionIDs) with zero edits to
    any existing assertion.
  • New tests added for gaps the old private lookup path had no way to exercise directly:
    • size-limit-exceeded degrades to a nil result with no error (does not take the cancellation-error
      path).
    • context cancellation, in two forms: already-canceled at entry, and canceled mid-fetch
      (deterministic, channel-coordinated, no sleeps) — plus a third subtest for
      context.DeadlineExceeded via a real timeout.
    • cancellation propagating through SymbolizePprof end-to-end.
    • concurrent fan-out: three mappings with distinct build IDs resolved concurrently
      (MaxDebuginfodConcurrency = 4), with a blocking-mock/channel handshake that forces genuine
      3-way overlap (a regression to serialized execution hangs the test rather than passing).
    • an invalid/malformed buildID degrades to a nil-per-miss result with zero calls to the
      debuginfod client or bucket, plus the new invalid_build_id metric label incrementing exactly
      once.
  • Coverage: 84.7% of statements in pkg/symbolizer.
  • go build ./... and make lint (golangci-lint run ./...) are both clean across the whole repo,
    not just this package.

Follow-ups

None deferred from review. Resolve's two adjacent string parameters (buildID, binaryName)
were flagged as a general API-ergonomics observation during review and deliberately left as-is —
the signature is pinned across the rest of this series and the one call site already threads both
arguments correctly.

jake-kramer and others added 2 commits July 9, 2026 16:16
…thout functions

The compaction symbols rewriter replaced every location of a partition
whose functions table is empty with a zero value: the branch that drops
line records for such partitions also skipped storing the location,
leaving InMemoryLocation{} in the lookup table. Partitions like this
are exactly the native, not-yet-symbolized profiles, where every
location is line-less, so compaction permanently destroyed the
addresses needed to symbolize them at read time — the source blocks
are deleted after compaction, so the loss is unrecoverable. Both the
v1 and v2 compactors rewrite symbols through this path.

Keep dropping the line records, which cannot reference any function,
but store the location so its address and mapping survive.

Introduced in #4051.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The symbolization end-to-end test asserted only the first successful
query, which happens before the pushed segment is compacted, so the
compaction rewrite of the block symbols was never exercised. Extend the
test past the first L0 compaction: filler blocks drive the age-based
batch flush, which is evaluated only when another block arrives at the
level, and the post-compaction assertions check the symbolized frames
without depending on location order, which compaction may change.

Without the rewriter fix, this fails with libfoo.so!0x0 fallback
frames as soon as the segment is compacted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a Resolver interface and a Resolve(ctx, buildID, binaryName,
addrs) method that pulls per-address symbol lookup out from behind
the pprof-only entry point. It gives future consumers a narrow
interface to depend on directly, without detouring through a
synthetic pprof profile just to get symbols resolved -- groundwork
for moving tree-shaped query symbolization out of the query rewrite.
SymbolizePprof now calls Resolve internally, and its output is
unchanged byte-for-byte on every existing fixture.

Two deliberate behavior notes:

- Resolve no longer synthesizes the "binary!0xaddr" fallback text
  itself. A nil/empty result slot means "unresolved," and rendering
  the fallback is now the caller's job: SymbolizePprof's per-mapping
  goroutine does it immediately after Resolve returns, so nothing
  downstream sees a difference.
- Resolve returns a non-nil error only when the context is done
  during resolution. That now surfaces as a real error out of
  SymbolizePprof, instead of silently returning a fallback-filled
  profile with a nil error. This is the one externally-visible
  behavior change here, and it only affects a path no existing
  fixture exercised.

Resolve also independently validates buildID with the same sanitizer
extractBuildID already applies, before doing any object-store or
debuginfod I/O. The only caller today already sanitizes first, so
this is a no-op in practice, but it's defense in depth now that
Resolve is a public entry point a future caller could reach directly
with unsanitized input.

Part of grafana/pyroscope-squad#487.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jake-kramer
jake-kramer force-pushed the symbolizer/resolve-seam branch from 29f06e1 to 436c24e Compare July 9, 2026 20:53
@jake-kramer
jake-kramer changed the base branch from main to fix/symdb-rewriter-lineless-locations July 9, 2026 20:54
@jake-kramer
jake-kramer marked this pull request as ready for review July 9, 2026 21:10
Base automatically changed from fix/symdb-rewriter-lineless-locations to main July 10, 2026 15:19
@jake-kramer
jake-kramer requested review from a team and simonswine as code owners July 10, 2026 15:19

@simonswine simonswine left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I think the error propagation is great, I am not too sure that is better to fail the whole flamegraph query, when a single symbolisation fails.

I think we should retain the user observable behaviour of using fallback symbols if we couldn't resolve some of the symbols.

Comment thread pkg/symbolizer/symbolizer.go Outdated
jake-kramer and others added 3 commits July 10, 2026 16:00
The debuginfod fetch flight is shared by every concurrent caller for a
build ID, but ran on the initiating caller's context, so one caller's
cancellation failed all waiters. Run the flight on a context detached
from the caller (values preserved, cancellation dropped); the HTTP
client timeout and bounded retries keep it from running unbounded.

Correspondingly, Resolve no longer classifies context errors propagated
from the fetch layer as its own: only the caller's context ending fails
the call, and every other failure - including another caller's
cancellation surfacing through a shared flight - degrades to unresolved
slots, which render as fallback symbols.

Suggested by simonswine in review.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… fallbacks

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread pkg/symbolizer/symbolizer.go

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e054fad. Configure here.

Comment thread pkg/symbolizer/debuginfod_client.go
…od fetch

Detaching the deduplicated debuginfod fetch from caller cancellation had cut
callers loose in both directions: a caller canceled mid-fetch could still
receive a successful result, and a canceled caller had to wait out the whole
detached fetch (HTTP timeout times retries) before returning. FetchDebuginfo
now waits on the shared result and the caller's context, whichever ends
first, and its doc comment owns the error contract: a done caller gets its
own context error immediately; any other error comes from the shared fetch
and says nothing about the caller's own context. Resolve folds its
post-fetch checks into a single own-context check instead of explaining
client internals at the call site.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jake-kramer
jake-kramer force-pushed the symbolizer/resolve-seam branch from e054fad to 441abfe Compare July 13, 2026 14:31

@simonswine simonswine left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing the context cancellation and looking into the fallback behaviour

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants