Prune unused passthrough columns from UNNEST output (opt-in)

[gortiz]

Summary

Adds an opt-in, default-off query option unnestColumnPruning for the multi-stage engine that prunes input/passthrough columns — notably the unnested source array — from the UNNEST output when nothing downstream references them.

Today, for a query like:

SELECT e.col1, u.s FROM e CROSS JOIN UNNEST(e.mcol1) AS u(s)

the UnnestNode output schema is the full Calcite Correlate row type [col1, mcol1, s], and UnnestOperator copies the entire input row (including the source array mcol1) into every one of the N exploded rows — only for a parent Project to immediately drop mcol1. For large arrays this needlessly widens every intermediate row (and serializes the array N times when an exchange sits between the UNNEST and the projecting operator).

The array expression is only needed in the operator's input (to evaluate the explode), never in its output, unless the user also selects it. With the flag on, the source array is no longer carried.

How it works

• UnnestNode carries a passthrough index map + prunedPassthrough flag. Legacy constructors default to not pruned (copy the whole input row), so existing behavior is unchanged.
• RelToPlanNodeConverter fuses the pruning into convertLogicalProject: when a Project sits directly above a Correlate/Uncollect (no wrapping correlate-filter), it computes which left columns the project actually references, builds a pruned UnnestNode (smaller output schema + passthrough map + recomputed element/ordinality indexes), and remaps that one project's InputRefs. It falls back to the current behavior in every other shape, and converts the correlate at most once.
• UnnestOperator honors the passthrough map, copying only retained columns (resolved to a primitive int[] so the per-row hot path stays allocation/box-free). When not pruned, it keeps the legacy whole-row System.arraycopy.

Backward compatibility / rolling upgrades

UnnestNode is serialized broker→server and the operator runs server-side, so this is a two-sided change. The proto fields (passthroughInputIndexes, prunedPassthrough) are additive:

• old broker → new server: fields absent ⇒ prunedPassthrough=false ⇒ legacy "copy whole row". Safe.
• new broker → old server: a smaller output schema would break an un-upgraded server. This is exactly why the option defaults to off — a new broker never emits the pruned schema until an operator explicitly enables it, which should only happen once the whole server fleet is upgraded. A future release can flip the default once a minimum server version is guaranteed.

Tests

• UnnestSqlPlannerTest — flag on/off, source-array-also-selected (no-op), WITH ORDINALITY, multiple arrays, zero-passthrough.
• PlanNodeSerDeTest — protobuf round-trip for pruned (non-sequential indexes + ordinality) and legacy UnnestNode.
• UnnestOperatorTest — pruned single-array, zero-passthrough, WITH ORDINALITY, multiple arrays; legacy path unchanged.
• UnnestIntegrationTest — end-to-end pruned-vs-default result equality across single/multi array, WITH ORDINALITY, zero-passthrough, and array-also-selected shapes.

Notes for reviewers

• UNNEST is only supported on the logical planner path, so the change is confined to RelToPlanNodeConverter + the node/operator + serde; the V2 physical path is untouched.
• Default-off; enabling it is a rolling-upgrade-ordering decision (servers before brokers).

[codecov-commenter]

Codecov Report

❌ Patch coverage is 4.54545% with 147 lines in your changes missing coverage. Please review.
✅ Project coverage is 37.26%. Comparing base (b27a3ad) to head (7af66f6).
⚠️ Report is 13 commits behind head on master.

Files with missing lines	Patch %	Lines
.../query/planner/logical/RelToPlanNodeConverter.java	1.76%	110 Missing and 1 partial ⚠️
...pache/pinot/query/planner/plannode/UnnestNode.java	0.00%	14 Missing ⚠️
...e/pinot/query/runtime/operator/UnnestOperator.java	0.00%	13 Missing ⚠️
...inot/query/planner/serde/PlanNodeDeserializer.java	0.00%	6 Missing ⚠️
.../pinot/query/planner/serde/PlanNodeSerializer.java	0.00%	3 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (b27a3ad) and HEAD (7af66f6). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (b27a3ad)	HEAD (7af66f6)
java-21	5	4
unittests1	1	0
unittests	2	1
temurin	5	4

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #18782       +/-   ##
=============================================
- Coverage     64.78%   37.26%   -27.52%     
+ Complexity     1309     1308        -1     
=============================================
  Files          3380     3380               
  Lines        209544   209683      +139     
  Branches      32797    32836       +39     
=============================================
- Hits         135746    78132    -57614     
- Misses        62870   124429    +61559     
+ Partials      10928     7122     -3806     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	37.26% <4.54%> (-27.52%)	⬇️
temurin	37.26% <4.54%> (-27.52%)	⬇️
unittests	37.25% <4.54%> (-27.52%)	⬇️
unittests1	?
unittests2	37.25% <4.54%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.