fix(substrait): roundtrip scalar subqueries in projections#23469
Draft
bvolpato wants to merge 1 commit into
Draft
fix(substrait): roundtrip scalar subqueries in projections#23469bvolpato wants to merge 1 commit into
bvolpato wants to merge 1 commit into
Conversation
3537b75 to
0f01204
Compare
0f01204 to
35f7501
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
With
datafusion.optimizer.enable_physical_uncorrelated_scalar_subquerydisabled, DataFusion rewrites a scalar subquery in a projection as a left join. The optimizer can remove itstruefilter before Substrait serialization. This leftJoinRel.expressionunset, even though Substrait requires it, and the consumer failed withPlan("join condition should not be empty").#18068 tried to fix the same roundtrip in the consumer, but also treated a conditionless inner
JoinRelas a cross join. Review questioned that behavior change, then the PR went stale. This version does not reinterpret an invalidJoinRel: the producer usesCrossRelfor conditionless inner joins and a literaltrueexpression for conditionless non-inner joins.What changes are included in this PR?
CrossRel.trueexpression for conditionless non-inner joins.Are these changes tested?
cargo test -p datafusion-substrait --features protoccargo clippy --all-targets --all-features -- -D warningsAre there any user-facing changes?
Scalar subqueries in projections now roundtrip when this optimizer option is disabled. No public API change.