Skip to content

fix: use build_join_schema for CrossJoinExec output schema metadata#23485

Open
gmhelmold wants to merge 1 commit into
apache:mainfrom
gmhelmold:fix/cross-join-schema-metadata
Open

fix: use build_join_schema for CrossJoinExec output schema metadata#23485
gmhelmold wants to merge 1 commit into
apache:mainfrom
gmhelmold:fix/cross-join-schema-metadata

Conversation

@gmhelmold

Copy link
Copy Markdown

Which issue does this PR close?

Closes #23434.

Rationale for this change

An aggregate over a pure cross join of two tables that carry the same schema-metadata key with different values fails at planning time:

Internal error: Physical input schema should be the same as the one converted from
logical input schema. Differences:
	- schema metadata differs: (physical) {"pandas": "RIGHT-SCHEMA"}
	                            vs (logical) {"pandas": "LEFT-SCHEMA"}

The logical plan builds a cross join as Join { join_type: Inner, .. } (LogicalPlanBuilder::cross_join), so its schema comes from build_join_schema, which merges input metadata left-wins. CrossJoinExec::new, however, merged metadata inline with an extend, which is right-wins. The two schemas therefore disagree on any conflicting metadata key, and the physical-vs-logical schema check rejects the plan.

CrossJoinExec is the only join operator that still hand-rolled its output schema. HashJoinExec, NestedLoopJoinExec, SortMergeJoinExec, SymmetricHashJoinExec and the piecewise merge join all already build their schema via build_join_schema. This aligns cross join with that precedent, in the same family as the union-metadata consistency fix #21127.

What changes are included in this PR?

  • CrossJoinExec::new now derives its output schema from build_join_schema(&left.schema(), &right.schema(), &JoinType::Inner) instead of the inline field-chain + right-wins metadata extend. The field list (left-then-right, no induced nullability — Inner sets force_nullable = false) is unchanged; only the metadata-merge order changes to left-wins, matching the logical schema.
  • A unit test (test_cross_join_metadata) mirroring joins::utils::test_join_metadata.
  • An end-to-end regression test (cross_join_with_conflicting_schema_metadata) reproducing the exact failure from CrossJoinExec merges schema metadata inconsistently with the logical plan #23434.

Are these changes tested?

Yes. Both tests fail on main with the errors quoted above and pass with the fix. The e2e's inputs (2-row left, 1-row right) also drive the JoinSelection swap path, exercising swap_inputs.

Are there any user-facing changes?

Yes — a bug fix. The output schema metadata of a cross join now follows left-wins on conflicting keys (consistent with the logical plan and with every other join operator). Field names, types, nullability and column order are unchanged. Aggregations over cross joins of metadata-carrying tables that previously errored now plan and execute.

@github-actions github-actions Bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CrossJoinExec merges schema metadata inconsistently with the logical plan

1 participant