Skip to content

fix: Make FilePruner work with constant columns#23388

Draft
mkleen wants to merge 2 commits into
apache:mainfrom
mkleen:file-prune-constant
Draft

fix: Make FilePruner work with constant columns#23388
mkleen wants to merge 2 commits into
apache:mainfrom
mkleen:file-prune-constant

Conversation

@mkleen

@mkleen mkleen commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

  • Currently FilePruner fails to prune PartitionedFile with a predicate of constant columns such as 5 < 100

What changes are included in this PR?

  • This extends the logic that a predicate with constant columns such as 5 < 100 is simplified to true when there is no dynamic filtering involved so the FilePruner can prune PartitionedFile

  • Original Test from Tests

  • More additional Tests

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions Bot added the datasource Changes to the datasource crate label Jul 8, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-datasource-parquet v54.0.0 (current)
       Built [  42.670s] (current)
     Parsing datafusion-datasource-parquet v54.0.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-datasource-parquet v54.0.0 (baseline)
       Built [  42.184s] (baseline)
     Parsing datafusion-datasource-parquet v54.0.0 (baseline)
      Parsed [   0.030s] (baseline)
    Checking datafusion-datasource-parquet v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.162s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  86.627s] datafusion-datasource-parquet
    Building datafusion-pruning v54.0.0 (current)
       Built [  37.039s] (current)
     Parsing datafusion-pruning v54.0.0 (current)
      Parsed [   0.012s] (current)
    Building datafusion-pruning v54.0.0 (baseline)
       Built [  37.528s] (baseline)
     Parsing datafusion-pruning v54.0.0 (baseline)
      Parsed [   0.012s] (baseline)
    Checking datafusion-pruning v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.084s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure method_parameter_count_changed: pub method parameter count changed ---

Description:
A publicly-visible method now takes a different number of parameters, not counting the receiver (self) parameter.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/method_parameter_count_changed.ron

Failed in:
  datafusion_pruning::FilePruner::new takes 5 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/6446decf518f29a21c68c2302ea8df8144d02676/datafusion/pruning/src/file_pruner.rs:58, but now takes 6 parameters in /home/runner/work/datafusion/datafusion/datafusion/pruning/src/file_pruner.rs:59
  datafusion_pruning::FilePruner::try_new takes 4 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/6446decf518f29a21c68c2302ea8df8144d02676/datafusion/pruning/src/file_pruner.rs:88, but now takes 5 parameters in /home/runner/work/datafusion/datafusion/datafusion/pruning/src/file_pruner.rs:91

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  75.583s] datafusion-pruning

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 8, 2026
}

#[tokio::test]
async fn test_prune_on_constant_columns() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the original test from #23272

use datafusion_physical_expr::planner::logical2physical;

#[test]
fn test_pruning_on_various_predicates() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More additional tests

}
// If there is no dynamic-filter-expression involved, convert constant expression
// such as `a > 100` to `true`/`false` so `PruningPredicate` can use to prune the file.
let predicate = if !self.tracking.contains_dynamic_filter() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works, but i am not sure if this the best place to do it. The expression will eventually end up in
https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/pruning_predicate.rs#L476
but the PhysicalExprSimplifier won't be applied because it's not a dynamic filter in this case.

@mkleen mkleen marked this pull request as ready for review July 8, 2026 13:26
@mkleen mkleen changed the title fix: Make FilePruner support constant columns fix: Make FilePruner work with constant columns Jul 8, 2026
@mkleen mkleen marked this pull request as draft July 8, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: FilePruner cannot prune constant columns

1 participant