fix: Make FilePruner work with constant columns#23388
Draft
mkleen wants to merge 2 commits into
Draft
Conversation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
mkleen
commented
Jul 8, 2026
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn test_prune_on_constant_columns() { |
mkleen
commented
Jul 8, 2026
| use datafusion_physical_expr::planner::logical2physical; | ||
|
|
||
| #[test] | ||
| fn test_pruning_on_various_predicates() { |
Contributor
Author
There was a problem hiding this comment.
More additional tests
mkleen
commented
Jul 8, 2026
| } | ||
| // If there is no dynamic-filter-expression involved, convert constant expression | ||
| // such as `a > 100` to `true`/`false` so `PruningPredicate` can use to prune the file. | ||
| let predicate = if !self.tracking.contains_dynamic_filter() { |
Contributor
Author
There was a problem hiding this comment.
This works, but i am not sure if this the best place to do it. The expression will eventually end up in
https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/pruning_predicate.rs#L476
but the PhysicalExprSimplifier won't be applied because it's not a dynamic filter in this case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
FilePrunerfails to prunePartitionedFilewith a predicate of constant columns such as5 < 100What changes are included in this PR?
This extends the logic that a predicate with constant columns such as
5 < 100is simplified totruewhen there is no dynamic filtering involved so theFilePrunercan prunePartitionedFileOriginal Test from Tests
More additional Tests
Are these changes tested?
Yes
Are there any user-facing changes?
No