perf(gfql): identity-stable _pl_nan_to_null + clean-cache (kills O(E) per-hop NaN re-scan on polars)

[lmeyerov]

Summary

Every hop() runs _coerce_input_formats → _pl_nan_to_null, which scanned is_nan().any() over every float column of the (unchanged) resident edge frame on every call. Repeated seeded hops on a resident graph — the seeded-Search / native-hop pattern — therefore grew O(E) with edge count on polars/polars-gpu, while pandas stayed flat. This is the dominant per-call cost that made polars/polars-gpu seeded LDBC Search grow with scale and lose to Kuzu.

Change (graphistry/Engine.py, _pl_nan_to_null)

• Probe an eager polars frame for real NaN once; return a clean frame UNCHANGED (restores the GFQL: #1658 index does not engage on polars/polars-gpu direct hops (frame-identity churn) #1726 identity guard that feat(gfql): add aggregate fast paths and final hardening #1731 reverted, so frame-identity caches like the feat(gfql): physical adjacency indexes for O(degree) seeded traversal #1658 CSR index keep engaging).
• Rewrite only the columns that genuinely carry NaN (values identical to the old unconditional fill_nan).
• Cache the id of clean frames so later calls skip the O(E) probe — recycle-safe via weakref.finalize (a rebound/new frame re-probes → NaN→null semantics unchanged).

Measured (dgx-spark)

Indexed polars g.hop on 8M edges with one float column: 33.8 ms (growing O(E)) → 0.20 ms FLAT (~140×, O(degree)), matching pandas. pandas unchanged (flat). polars no-float hop still grows mildly (0.17→0.91 ms) — a smaller, separate per-call cost, out of scope here.

Validation

• Regression: 184 passed — full 4-engine test_index.py parity + test_engine_coercion.py. The one failure (test_chain_dask_edges) is pre-existing on clean master (RAPIDS-image dask issue; verified on the unpatched checkout).
• New TestPlNanCleanCache: NaN-present cleaned to null; clean float frame returned same-object + cached; distinct frames don't cross-contaminate (all 4 pass).

Relationship to #1733

Subsumes PR #1733 (the #1726 identity-stable reland) — this does the identity-stable behavior and adds the clean-cache. #1733 can be closed in favor of this, or this rebased on it.

Impact

Makes polars/polars-gpu seeded Search O(degree) on float-column (i.e. real) graphs — the contained lever toward a 4-engine LDBC Search win. Broadly speeds all polars Search that re-hops a resident graph.

🤖 Generated with Claude Code

[lmeyerov]

STACK SEPARATION (2026-07-19, for manual review):

READY FOR REVIEW (stable, low-churn, land bottom-up):

1. perf(gfql): identity-stable _pl_nan_to_null + clean-cache (kills O(E) per-hop NaN re-scan on polars) #1735 (this) — nan-clean-cache. Tiny, self-contained; closes fix(gfql): restore identity-stable _pl_nan_to_null (regression from #1731) #1733 (subsumed). Lint-fixed (py3.8 Set[int]).
2. perf(gfql): engage #1658 index for seeded gfql()/Cypher chains on ALL 4 engines (incl. typed edges) #1734 — typed-edge index engagement, 4-engine, 138 index tests. Lint-fixed (mypy narrow + cast).
3. feat(gfql): polars-native connected OPTIONAL MATCH (fixes pandas-ism crash + unseeded-arm dispatch + polars join twin) #1736 — polars-native connected OPTIONAL MATCH (crash fix → native arm+join → CASE-null → arm pruning). Oracle-pinned tests.
4. feat(gfql): polars whole-entity aggregation Cypher — HAS_label disambiguation + identity-key + key_prefixes (LDBC IC4 native) #1737 — polars whole-entity aggregation (HAS_label disambiguation ported at pandas' exact call site + identity-key + key_prefixes). Both narrowing branches test-pinned after self-review caught 2 parity divergences.

EXPERIMENTAL (high-churn, do NOT land yet — iterating with the benchmark campaign):
5. #1738 — fast-path caches (slices 1-2; slice 3 lean gather-combine in design, spec in plans/gfql-benchmark-numbers/plan.md). Perf-cache design may still move as slices land.

pyg-bench #90 (runner/config migration) is independent and review-ready.

Whole-stack validation on dgx at the #1738 tip after today's lint fixes + cascade rebase: 362 passed, 1 pre-existing failure (test_chain_dask_edges — fails on master too, unrelated). Each PR is content-identical to its pre-rebase review state modulo the two lint commits.

[lmeyerov]

type + factor otu