Skip to content

feat: add metrics tracking for file sinks and update orchestration fo…#23490

Open
CadyLinn wants to merge 1 commit into
apache:mainfrom
CadyLinn:feat/csv-json-sink-metrics
Open

feat: add metrics tracking for file sinks and update orchestration fo…#23490
CadyLinn wants to merge 1 commit into
apache:mainfrom
CadyLinn:feat/csv-json-sink-metrics

Conversation

@CadyLinn

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

CSV and JSON sinks do not currently expose write metrics through DataSinkExec, while the Parquet sink exposes rows_written, bytes_written, and elapsed_compute.

This change provides consistent metrics for CSV and JSON sinks and centralizes the common file sink metrics wiring to reduce duplication across file formats.

What changes are included in this PR?

  • Add a shared FileSinkMetrics implementation for rows_written, bytes_written, and elapsed_compute.
  • Use the shared metrics implementation for Parquet, CSV, and JSON sinks.
  • Track rows and serialized bytes in the common stateless write orchestration.
  • Preserve the existing spawn_writer_tasks_and_join API and add an opt-in metrics variant.
  • Add SQL logic test coverage for EXPLAIN ANALYZE COPY with CSV and JSON output.

Are these changes tested?

Yes. The affected datasource unit tests and the copy.slt SQL logic test pass.

cargo test -p datafusion-datasource \
  -p datafusion-datasource-csv \
  -p datafusion-datasource-json \
  -p datafusion-datasource-parquet --lib
cargo test -p datafusion-sqllogictest \
  --test sqllogictests -- copy.slt --complete

The affected crates also pass targeted Clippy checks with warnings denied.

Are there any user-facing changes?

Yes. EXPLAIN ANALYZE COPY now reports rows_written, bytes_written, and elapsed_compute for CSV and JSON sinks, consistent with the existing Parquet sink metrics.

There are no breaking public API changes.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate labels Jul 12, 2026
@CadyLinn CadyLinn marked this pull request as ready for review July 12, 2026 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement sink metrics but CSV/JSON

1 participant