fix: stringify JSON search sidebar filters

[mfroembgen]

Summary

• Detect JSON-backed source columns and render sidebar filter keys as ClickHouse string expressions, for example toString(ResourceAttributes.\k8s`.`namespace`.`name`), instead of JSON map access or a fixed .:String` suffix.
• Canonicalize stale URL/saved sidebar filters that still use JSON map access once source columns load.
• Reuse the same JSON string rendering for search URL filters, sidebar value queries, load-more queries, and distribution queries.
• Prioritize active and pinned fields when fetching initial facet values so selected combinations show their matching options before Load more.
• Add unit coverage for JSON filter serialization, hydration, stale persisted filter canonicalization, and metadata value/distribution SQL rendering.

Why toString(...)

Testing against ClickHouse 26.5.1.882 with JSON(max_dynamic_types=8, max_dynamic_paths=64) columns showed that .:String is only safe for JSON paths whose active dynamic type is actually String. Integer, boolean, and mixed-type paths can silently return no values with .:String, while toString(<JSON path>) returned usable values for all of these real path shapes:

• ResourceAttributes.k8s.namespace.name (string)
• ResourceAttributes.cloud.account.id (integer)
• LogAttributes.endOfBatch (boolean)
• LogAttributes.level (mixed integer/string)

The same toString(...) expression also worked for distribution/grouping queries.

Manual verification

• Verified directly against ClickHouse 26.5.1.882, where ResourceAttributes and LogAttributes are JSON(max_dynamic_types=8, max_dynamic_paths=64) columns.
• Metadata-style query over a live data window returned non-empty values for string, int, bool, and mixed JSON paths with toString(...).
• Distribution-style grouping returned values for toString(ResourceAttributes.\cloud`.`account`.`id`)andtoString(LogAttributes.`endOfBatch`)`.
• Verified a local dashboard flow against a live API: stale ResourceAttributes['k8s.namespace.name'] filters were canonicalized, results loaded without the ClickHouse arrayElement JSON error, Show Distribution loaded, and selected/pinned facets refreshed with matching values plus Load more.

Automated verification

• mise exec node@22.16.0 -- node .yarn/releases/yarn-4.13.0.cjs lint:fix
• mise exec node@22.16.0 -- node ../../.yarn/releases/yarn-4.13.0.cjs ci:lint in packages/common-utils
• mise exec node@22.16.0 -- node ../../.yarn/releases/yarn-4.13.0.cjs ci:unit in packages/common-utils
• mise exec node@22.16.0 -- node .yarn/releases/yarn-4.13.0.cjs workspace @hyperdx/app jest packages/app/src/components/__tests__/DBSearchPageFilters.test.tsx packages/app/src/components/DBSearchPageFilters/utils.test.ts packages/app/src/__tests__/searchFilters.test.ts --runInBand --coverage=false
• mise exec node@22.16.0 -- bash -lc 'corepack enable >/dev/null 2>&1 || true; make dev-e2e FILE=filter-key-edge-cases' (18 passed)

References

Fixes #2549.
Related to #2482 and #2537.

[changeset-bot]

🦋 Changeset detected

Latest commit: f58f69f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/app	Patch
@hyperdx/common-utils	Patch
@hyperdx/api	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@mfroembgen is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

This PR fixes JSON-backed sidebar filters by replacing the unreliable .:String typed-subcolumn syntax with toString(col.path) expressions that work correctly for string, integer, boolean, and mixed-type JSON paths in ClickHouse 26.5. It also canonicalizes stale persisted filters (bracket form, typed subcolumn form) to the new expression format when source columns load, and introduces a generation counter to discard stale out-of-order load-more responses.

• renderJsonStringExpression / parseRenderedJsonStringExpression — new pair of export functions in common-utils that encode/decode toString(col.\seg`.`seg`)expressions (with backslash and backtick escaping) and a specialJSONExtractString(toJSONString(col), '')form for empty JSON keys; allrenderJsonStringSubcolumn` call sites are migrated.
• canonicalizeFilterQuery — rewrites stale URL/saved filters to toString(...) form on load, guarded by a round-trip check to avoid touching compound SQL predicates; a canonicalizedFiltersSyncRef prevents the prevSearched effect from triggering an unwanted form reset.
• FilterGroup expansion overhaul — separates isDefaultExpanded-transition expansion from auto-expand-for-Load-More, honours user manual toggles via hasUserToggledExpansionRef, and clears loadMoreLoadingKeys atomically on scope change.

Confidence Score: 4/5

Safe to merge for the common case; one narrow routing defect in getAllKeyValues affects Load More for empty-string JSON keys in MV-backed deployments only.

The empty-key JSON expression assigns mapKey: '' (falsy), causing the MV dispatch to intercept it with rollupKey: '' and bypass the rawTableExpression fallback. In MV-backed setups, Load More for ResourceAttributes[''] returns zero results. The defect is limited to empty-string JSON attribute keys, which are rare in practice.

Files Needing Attention: packages/common-utils/src/core/metadata.ts — the getAllKeyValues routing loop around lines 2405–2481, specifically the empty-key branch where mapKey: '' is falsy.

Important Files Changed

Filename	Overview
packages/common-utils/src/core/metadata.ts	Replaces .:String typed-subcolumn rendering with toString(col.path) expressions; adds renderJsonStringExpression, parseRenderedJsonStringExpression, stripClickHouseJsonTypeSuffix, and routes getAllKeyValues through the new parsed form. Empty-key JSON expression (JSONExtractString(...)) has a falsy mapKey that can be intercepted by the MV path before the rawTableExpression branch in MV-backed setups.
packages/app/src/components/DBSearchPageFilters/utils.ts	Adds jsonColumns option to toQuotedClickHouseKeyExpression so JSON-backed sidebar keys render as toString(col.path) instead of bracket-map access; imports and delegates to the new common-utils helpers; removes the local quoteIdentifierIfNeeded in favour of the shared export.
packages/app/src/searchFilters.tsx	Adds canonicalizeFilterQuery that rewrites stale bracket-form or typed-subcolumn JSON filters to toString(...) form once source columns load, with a round-trip guard to avoid dropping compound SQL predicates; threads jsonColumns through escapeFilterStateKeys and useSearchPageFilterState.
packages/app/src/DBSearchPage.tsx	Moves inputSourceTableConnection earlier to share with the new useJsonColumns call; adds canonicalizedFiltersSyncRef to prevent the prevSearched effect from resetting the form on canonicalization-only URL updates; wires jsonColumns through to useSearchPageFilterState.
packages/app/src/components/DBSearchPageFilters/hooks.ts	Threads jsonColumns through useFetchFacets into key-escaping and load-more paths; adds loadMoreGenerationRef to discard stale out-of-order load-more responses and clears loadMoreLoadingKeys on scope change.
packages/app/src/components/DBSearchPageFilters.tsx	Threads jsonColumns into sqlKey rendering; refactors FilterGroup expansion logic to distinguish isDefaultExpanded transitions from auto-expand-for-Load-More, respecting user manual toggles via hasUserToggledExpansionRef.
packages/app/src/tests/searchFilters.test.ts	Adds thorough unit coverage for JSON filter serialization, hydration, and canonicalization (idempotency, bracket form, typed subcolumn form, empty key, compound predicates, non-JSON passthrough).
packages/common-utils/src/tests/metadata.test.ts	Updates existing JSON tests from .:String to toString(...) form; adds coverage for backslash escaping, empty-key JSONExtractString form, MV and raw-table routing for rendered JSON expressions, and renderJsonStringExpression/parseRenderedJsonStringExpression round-trips.
packages/app/src/components/tests/DBSearchPageFilters.test.tsx	Adds UI-level tests for empty-facet Load More visibility, user-toggle persistence, default-expanded transitions, and stale-result rejection during concurrent load-more requests.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Sidebar filter key] --> B{Base column in jsonColumns?}
    B -- Yes --> C[renderJsonStringExpression
toString col.path]
    B -- No --> D[bracket form col key]
    C --> E[Persisted in URL/DB]
    D --> E
    E --> F{canonicalizeFilterQuery on load}
    F -- Already toString form --> G[idempotent: no change]
    F -- Stale bracket or .:String form --> H[parse → clean → re-escape]
    H --> I[toString expression written back]
    I --> J[getAllKeyValues]
    J --> K{parseRenderedJsonStringExpression}
    K -- Non-empty key --> L[correct MV or raw-table routing]
    K -- Empty key mapKey=empty falsy --> M[⚠ MV intercepts rollupKey=empty
raw table JSONExtractString bypassed]
    K -- Not a JSON expression --> N[parseKeyPath bracket or native column]

Loading

_{Reviews (35): Last reviewed commit: "fix: stringify JSON search sidebar filte..." | Re-trigger Greptile}

[github-actions]

🔴 P0/P1 -- must fix

• packages/app/src/components/DBSearchPageFilters/utils.ts:25 -- cleanClickHouseExpression strips backticks but never reverses the backslash-doubling quoteJsonPathSegment applies, so canonicalizeFilterQuery is non-idempotent for a JSON path segment containing a backslash or backtick; the DBSearchPage memo/effect then re-setValues an ever-growing filter each render and crashes with "Maximum update depth exceeded".
  • Fix: Make cleanClickHouseExpression the exact inverse of quoteJsonPathSegment (halve doubled backslashes and undouble backticks), or reuse the metadata-side parseRenderedJsonStringExpression, so escape∘unescape reaches a fixed point.
  • _{adversarial, correctness, maintainability, kieran-typescript}

🟡 P2 -- recommended

• packages/app/src/DBSearchPage.tsx:1315 -- The canonicalization effect snapshots live form state via getValues() into setSearchedConfig, so when useJsonColumns resolves asynchronously after the user has typed an unsubmitted where/select, it commits and executes that draft and rewrites the URL.
  • Fix: Update only the filters field from the already-searched searchedConfig, not the live form, so canonicalization cannot pick up unsubmitted edits.
  • _{adversarial, julik-frontend-races}
• packages/app/src/components/DBSearchPageFilters.tsx:945 -- The shouldExpandForLoadMore effect only ever calls setExpanded(true) and re-fires on every false→true transition (e.g. live-tail refetch toggling optionsLoading while extraFacetKeys resets), re-expanding a facet group the user just collapsed.
  • Fix: Make auto-expand one-shot per key (track already-auto-expanded keys in a ref) so it never re-asserts after a manual collapse.
  • _{adversarial, julik-frontend-races}
• packages/common-utils/src/core/metadata.ts:185 -- The new security-sensitive JSON-path parsers (parseRenderedJsonStringExpression, parseClickHouseIdentifier, parseRenderedJsonPath, stripClickHouseJsonTypeSuffix) are only exercised through happy-path callers, with no direct negative or adversarial-escaping coverage.
  • Fix: Add direct tests for malformed inputs (unterminated toString(/backtick, unquoted segments, empty path) and round-trip cases with doubled backticks and backslashes.
  • _{kieran-typescript, testing}

🔵 P3 nitpicks (6)

• packages/common-utils/src/core/metadata.ts:2217 -- parseRenderedJsonStringExpression uses bracket indexing innerExpression[parsedColumn.endIdx] while the rest of the module uses charAt(), relying on noUncheckedIndexedAccess being off.
  • Fix: Use charAt() for consistency and undefined-safety.
• packages/common-utils/src/core/metadata.ts:60 -- quoteJsonPathSegment was newly exported but has no consumer outside metadata.ts, and four low-level quoting helpers now widen the published @hyperdx/common-utils API surface.
  • Fix: Keep quoteJsonPathSegment module-private and confirm the other helper exports are intended public contract.
• packages/app/src/components/DBSearchPageFilters/utils.ts:5 -- The imported quoteIdentifierIfNeeded unquotes and escapes backslashes before quoting, diverging from the removed SqlString.escapeId version for already-quoted or backslash-containing flat column names.
  • Fix: Add a regression assertion that flat-column quoting still emits identical SQL for special-character column names.
• packages/app/src/components/DBSearchPageFilters/hooks.ts:140 -- Call sites pass { jsonColumns: jsonColumns ?? [] } even though toQuotedClickHouseKeyExpression already defaults options.jsonColumns ?? [] internally, duplicating the empty-array default in ~6 places.
  • Fix: Pass jsonColumns directly and let one layer own the default.
• packages/app/src/components/DBSearchPageFilters/hooks.ts:130 -- The escapedKeysToFetch memo gates only on isColumnsLoading, so if useColumns resolves before useJsonColumns (still []), JSON sub-keys are briefly escaped as invalid map-access SQL until the memo re-runs.
  • Fix: Also gate the memo on jsonColumns having loaded before fetching JSON-backed keys.
• packages/common-utils/src/core/metadata.ts:1695 -- getKeyValues rollup parsing only recognizes the new toString(...) form, so legacy .:String keys that were never canonicalized fall through parseKeyPath and mis-split into rollupColumn/rollupKey, degrading facet value suggestions.
  • Fix: Recognize and strip the legacy .:String typed-subcolumn form in the keyExpressions parse step.

---

Reviewers (8): correctness, security, adversarial, julik-frontend-races, kieran-typescript, testing, maintainability, api-contract.

Testing gaps:

• DBSearchPage canonicalizedFilters memo/effect has no coverage of its loop-avoidance guard or its non-overwrite behavior.
• canonicalizeFilterQuery non-sql early bail, length-mismatch bail, dateTimeColumns argument, and ordering-difference skip paths are untested.
• No adversarial round-trip tests for JSON path segments containing backslashes, backticks, closing parens, or newlines.

[github-actions]

Deep Review

Scope: PR #2551 — stringify JSON search sidebar filters (base 01508d1d). ~500 lines of executable TypeScript across common-utils/core/metadata.ts, searchFilters.tsx, DBSearchPageFilters/{utils,hooks}.ts, DBSearchPage.tsx, DBSearchPageFilters.tsx, plus substantial added unit coverage.

Intent: Render JSON-backed source column filter keys as ClickHouse string expressions (toString(col.\a`.`b`), or JSONExtractString(toJSONString(col), '')for empty paths) instead of map access or a fixed.:String` suffix; canonicalize stale persisted filters once columns load; reuse the rendering across value/load-more/distribution queries; prioritize active/pinned facets; add tests.

✅ No critical issues found. The escape/decode pair (quoteJsonPathSegment backslash+backtick doubling ↔ parseClickHouseIdentifier) round-trips symmetrically, so the UNSAFE_RAW_SQL distribution/value paths in metadata.ts are not a new injection vector; column names originate from table schema, not free user input. The removed renderJsonStringSubcolumn / preserveStringTypeSuffix / JSON_STRING_TYPE_SUFFIX symbols have no stale callers in packages/, and KubernetesDashboardPage.tsx builds its .:String expressions by hand (not via the changed renderer), so it is unaffected by the signature/output change.

🟡 P2 — recommended

• packages/app/src/searchFilters.tsx:447 — canonicalizeFilterQuery feeds setValue('filters', …) in DBSearchPage.tsx, whose result flows back through useWatch into the canonicalizedFilters memo; convergence depends entirely on the function being a fixed point, and the only guard against re-firing every render is the JSON.stringify null-check.
  • Fix: Add a unit test asserting canonicalizeFilterQuery is idempotent (second pass byte-identical to the first) for JSON filters, including conditions with irregular whitespace and quoting.
• packages/common-utils/src/core/metadata.ts:786 — the empty-path branch emits JSONExtractString(toJSONString(<column>), ''), which materializes the whole JSON column per row and can scan-cost more than a targeted path on large tables.
  • Fix: Confirm this branch is only reachable for genuinely empty keys and document (or cheapen) the whole-column serialization cost.

🔵 P3 nitpicks (4)

• packages/common-utils/src/core/metadata.ts:629 — stripClickHouseJsonTypeSuffix and parseRenderedJsonStringExpression are heuristic string matchers; a JSON leaf key that legitimately ends in .:<TypeLikeToken> or an expression shaped like toString(...) can be silently reclassified, altering the emitted filter.
  • Fix: Add coverage for these look-alike keys and confirm the pre-existing .:String ambiguity boundary is preserved.
• packages/app/src/components/DBSearchPageFilters/utils.ts:350 — repeated jsonColumns ?? [] defaulting is threaded through several call sites; normalize the default once at the hook boundary.
  • Fix: Default jsonColumns to [] where the value enters the module and drop the per-call fallbacks.
• packages/common-utils/src/core/metadata.ts:641 — the hand-rolled ClickHouse identifier parser (parseClickHouseIdentifier/parseRenderedJsonPath) adds parser-maintenance surface coupled by literal toString(/JSONExtractString(toJSONString( prefixes to the renderer.
  • Fix: Keep encode and decode colocated with a comment cross-referencing each other so the literal contract stays visible.
• packages/app/src/components/DBSearchPageFilters.tsx:912 — hasUserToggledExpansionRef is never reset, so once a user toggles a mounted FilterGroup the shouldExpandForLoadMore auto-expansion will not re-fire for that instance even after the filter scope changes.
  • Fix: Reset the toggle ref when the group's key/scope changes if re-expansion on new empty results is intended.

---

Reviewers (12): correctness, security, adversarial, testing, maintainability, project-standards, api-contract, kieran-typescript, julik-frontend-races, performance, agent-native, learnings-researcher.

Testing gaps:

• Idempotency of canonicalizeFilterQuery (guards against a render/setValue loop).
• Explicit round-trip assertions for JSON path segments containing backticks and backslashes (encode↔decode appears correct; lock it with tests).
• A stale-response test for the loadMoreGenerationRef generation check (drop results whose generation no longer matches; ensure loadMoreLoadingKeys never leaks a stuck spinner).

[greptile-apps]

Empty-key JSON expression bypasses rawTableExpression in MV-backed setups

When keyExpressions contains JSONExtractString(toJSONString(ResourceAttributes), ''), parseRenderedJsonStringExpression returns { column: 'ResourceAttributes', key: '' }, so mapKey is assigned '' (an empty string, which is falsy). The routing loop then:

1. Skips the map-text-index branch (if (key.mapKey) → false)
2. Falls into the native-text-index else branch (no match, no continue)
3. Reaches the MV check — if ResourceAttributes is in metadataMVs, it pushes rollupKey: '' into the MV query options and continues, never reaching the rawTableExpression branch below

The MV has no entry for an empty rollup key, so the query returns zero results. In a MV-backed deployment, "Load more" for a sidebar facet canonicalized from ResourceAttributes[''] silently returns nothing instead of executing the correct JSONExtractString(toJSONString(ResourceAttributes), '') scan.

The fix is to guard on key.rawTableExpression before the MV check for the empty-key case, or to use mapKey: renderedJsonKey.key || undefined so the empty-key path falls straight through to the raw table branch.