Skip to content

Add support for huggingface datasets to vortex-python - #8733

Open
robert3005 wants to merge 2 commits into
developfrom
rk/hfd
Open

Add support for huggingface datasets to vortex-python#8733
robert3005 wants to merge 2 commits into
developfrom
rk/hfd

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Add IterableDataset implementation backed by vortex files. Majority of the logic in this pr deal with adapting hugging face options into compatible vortex options

@robert3005 robert3005 added the changelog/feature A new feature label Jul 13, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 13, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 17.05%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
✅ 1891 untouched benchmarks
⏩ 3 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime cuda/bitpacked_u8/unpack/3bw[100M] 351.5 µs 300.3 µs +17.05%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/hfd (f0a261c) with develop (b15394a)

Open in CodSpeed

Footnotes

  1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@vortex-data vortex-data deleted a comment from github-actions Bot Jul 27, 2026
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant