Prefetch Parquet row groups with a bounded I/O budget#23492
Conversation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/parquet-pushdecoder-prefetch (4d02e1e) to 2880e10 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/parquet-pushdecoder-prefetch (4d02e1e) to 2880e10 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/parquet-pushdecoder-prefetch (4d02e1e) to 2880e10 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/parquet-pushdecoder-prefetch (4d02e1e) to 2880e10 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/parquet-pushdecoder-prefetch (4d02e1e) to 2880e10 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Which issue does this PR close?
Rationale for this change
Sequential Parquet row-group reads are expensive when object-store requests have meaningful latency. Batching projected ranges can hide much of that latency without eagerly decoding data.
What changes are included in this PR?
datafusion.execution.parquet.prefetch_sizebyte budget, disabled by default.Results
With a 20 MiB budget, simulated object-store latency is 2-3x (see results brlow).
With latency disabled, performance remained effectively neutral: 1.02x for TPC-H and 1.01x for ClickBench. Raising the budget to 100 MiB provided no further aggregate benefit.
Are these changes tested?
Are there any user-facing changes?
Users can opt in by setting
datafusion.execution.parquet.prefetch_size; the default remains disabled.