Skip to content

test(spark): characterize VortexBatchExec input partition planning#8859

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-batch-exec-tests
Open

test(spark): characterize VortexBatchExec input partition planning#8859
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-batch-exec-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

VortexBatchExec (java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexBatchExec.java)
turns the scan's resolved paths into Spark InputPartitions, and it currently has no
direct test coverage. Its planning contract is easy to break silently: each .vortex
file must become exactly one VortexFilePartition, each partition must carry the
Hive-style partition values parsed from its own path (not a neighbour's), and the
read schema and format options must survive into every partition — these are what the
executor-side reader relies on to materialize partition columns correctly. This
continues the characterization-test series from #8770, #8782, #8783, #8850, and #8857.

What changes are included in this PR?

A new VortexBatchExecTest with 5 tests covering planInputPartitions for explicit
.vortex paths (which never touch native directory listing, keeping this a pure unit
test):

  • one VortexFilePartition per file, each holding exactly one path;
  • partitions preserve the input path order;
  • the requested read schema is carried into every partition;
  • Hive-style partition values are parsed from each file's own path (year=2024/month=01
    vs year=2025/month=02 vs an unpartitioned path yielding an empty map);
  • format options are propagated to every partition.

This is a test-only change; no production code is modified. All 5 tests pass
locally on both matrix legs (:vortex-spark_2.12:test and :vortex-spark_2.13:test
with --tests 'dev.vortex.spark.read.VortexBatchExecTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

@robert3005 robert3005 added the changelog/chore A trivial change label Jul 21, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 21, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.17%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 44 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation true_count_vortex_buffer[128] 580.6 ns 522.2 ns +11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:spark-batch-exec-tests (76202cd) with develop (b7b01d3)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@jackylee-ch
jackylee-ch force-pushed the spark-batch-exec-tests branch from 76202cd to 3ac4c0e Compare July 21, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants