Skip to content

fix(spark): include the offending path when schema inference finds no Vortex files#8858

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-infer-schema-error-msg
Open

fix(spark): include the offending path when schema inference finds no Vortex files#8858
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-infer-schema-error-msg

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

When VortexDataSourceV2.inferSchema is pointed at a directory that contains no
.vortex files, it currently throws
RuntimeException("UNABLE_TO_INFER_SCHEMA format: vortex"). The message does not say
which path failed, so with multi-path reads (or a typo'd location) users have nothing
to go on. This is the same kind of diagnosability gap as #8759 (unsupported ColumnVector
type error message).

What changes are included in this PR?

  • inferSchema now throws IllegalArgumentException with a message that names the
    offending path and suggests the likely fixes:
    Unable to infer schema for vortex: no .vortex files found under path <path>. Check that the path is correct and contains at least one Vortex file, or provide an explicit schema.
  • The @throws javadoc on inferSchema is updated to match.
  • A new VortexDataSourceInferSchemaTest reads an empty temp directory through the real
    spark.read().format("vortex").load(...) path and asserts the failure message names
    the path (walking the cause chain, since Spark may wrap data source exceptions).

Passing locally on both matrix legs (:vortex-spark_2.12:test and
:vortex-spark_2.13:test with --tests 'dev.vortex.spark.VortexDataSourceInferSchemaTest').

What APIs are changed? Are there any user-facing changes?

The exception type thrown by inferSchema for the no-files case changes from a bare
RuntimeException to IllegalArgumentException (still a RuntimeException subtype),
and the message text changes as described. No other behavior changes.

@robert3005 robert3005 added the changelog/chore A trivial change label Jul 20, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.17%

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 44 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation true_count_vortex_buffer[128] 580.6 ns 522.2 ns +11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:spark-infer-schema-error-msg (94dd273) with develop (b7b01d3)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

… Vortex files

Signed-off-by: jackylee <qcsd2011@gmail.com>
@jackylee-ch
jackylee-ch force-pushed the spark-infer-schema-error-msg branch from 94dd273 to 9884816 Compare July 21, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants