feat(vortex-file): gate the file writer behind editions#8877
Draft
joseph-isaacs wants to merge 2 commits into
Draft
feat(vortex-file): gate the file writer behind editions#8877joseph-isaacs wants to merge 2 commits into
joseph-isaacs wants to merge 2 commits into
Conversation
joseph-isaacs
marked this pull request as draft
July 21, 2026 12:41
The session is the source of truth for what a writer may emit: EditionSession now stores the set of enabled editions alongside the declarations — enable() (validated, at most one edition per family), enabled(), and enabled_encodings(), the resolved union of the enabled editions' encoding sets. register_default_editions declares the editions and enables the default write editions: core2026.07.0, plus the newest unstable draft under the `unstable_encodings` feature. The writer resolves its allow set from the session at write time: - The flat leaf writer's normalize-with-error validation fails the write for any encoding outside the session's enabled editions. - The writer's pre-populated ArrayContext is seeded from the same set. - Compression is gated at scheme-selection time: the new required Scheme::produced_encodings method declares the array encodings a scheme may emit (empty means only canonical arrays), and CascadingCompressor::with_enabled_editions_gating skips schemes whose outputs are not all within the enabled editions, resolved from the execution context per compression. Sessions with no enabled editions fall back to the static ALLOWED_ENCODINGS registry, and explicit with_allow_encodings sets still override everything. Experimental patches remain a runtime opt-out: Patched is only writable when the experimental environment variable is set, mirroring when compression produces it. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2
joseph-isaacs
force-pushed
the
claude/writer-editions-gating-8qy1fv
branch
from
July 21, 2026 13:54
bf7ca36 to
2da25df
Compare
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
- Qualify the DEFAULT_WRITE_EDITIONS intra-doc link in the editions module docs: links in the module's inner docs resolve at the facade's `pub mod` site, so unqualified links break rustdoc under -D warnings. Also move the stray "Logical Vortex data types." doc line back onto the dtype module. - Update the file-size golden values in the Python write-options doctests: the writer's pre-populated ArrayContext is now seeded from the session's enabled editions, which changes the footer's encoding list and shrinks the written files by 40 bytes. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2
Contributor
Benchmarks: Vortex queries 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.127x ❌, 0↑ 2↓)
datafusion / parquet (1.050x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.160x ❌, 0↑ 2↓)
duckdb / parquet (1.034x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Follow-up to the editions implementation (#8871): the edition declarations existed but nothing enforced them. This PR gates the file writer behind editions. The session is the source of truth: it stores the set of enabled editions alongside the declarations, and exposes the resolved set of all enabled encodings. The default session enables
core2026.07.0(EditionId::new("core", 2026, 7, 0)), so by default:What changes are included in this PR?
vortex-edition:EditionSessionnow stores the enabled editions next to the declarations:enable()(must be declared; at most one edition per family),enabled(), andenabled_encodings()— the resolved, sorted union of the enabled editions' encoding sets (Nonewhen nothing is enabled). The first-party declarations stay in thevortexfacade; this crate remains types + session variable + test harness.vortex(facade):register_default_editionsnow also enablesDEFAULT_WRITE_EDITIONS(core2026.07.0, plus the newestunstabledraft when theunstable_encodingsfeature opts in) on the session. New tests: a golden pin of the frozencore2026.07.0encoding set, per-edition harness validation, subset properties, and a check that every default compression scheme's produced encodings are covered by the default write editions.vortex-compressor: new requiredScheme::produced_encodings(&self) -> Vec<ArrayId>trait method declaring the array encodings a scheme may emit (empty = only canonical arrays; cascaded children are covered by their own schemes).CascadingCompressor::with_enabled_editions_gatingmakes scheme selection resolve the session's enabled encodings from the execution context at compression time and skip schemes whose outputs are not all allowed; sessions with no enabled editions are not gated. Unit-tested with a synthetic edition/scheme.vortex-btrblocks: every scheme declares its produced encodings (including the env-gatedPatchedoutput of BitPacking/ALP).BtrBlocksCompressorBuildergainsgate_by_enabled_editions()(session gating pass-through) andretain_allowed_encodings()(static filtering for explicit allow sets).vortex-layout:FlatLayoutStrategy::with_enabled_editions_gatingresolves the allowed encodings from the session's enabled editions atwrite_streamtime — an encoding outside them fails the write via the existing normalize-with-Errorvalidation — falling back to the explicitallowed_encodingsset when the session enables no editions.vortex-file: the defaultWriteStrategyBuilderwires all of this up: flat writer gated by enabled editions with the staticALLOWED_ENCODINGSregistry as fallback, and both the data and stats compressors gated at compression time. An explicitwith_allow_encodingsset overrides the session gate everywhere (unchanged behavior forvortex-jni/vortex-parquet-variant). The writer's deterministic pre-populatedArrayContextlikewise comes from the session's enabled encodings. Experimental patches remain a runtime opt-out:Patchedis only writable when the experimental environment variable is set, mirroring when compression produces it.Checks run (after rebasing onto
developat #8871):cargo nextest run -p vortex-edition -p vortex-compressor -p vortex-btrblocks -p vortex-layout -p vortex-file -p vortex(411 passed; also with--all-featuresfor file/btrblocks/compressor),cargo test --docfor the touched crates,cargo +nightly fmt --all, andcargo clippy --all-targets --all-featuresfor the touched crates plusvortex-tensor,vortex-jni, andvortex-parquet-variant(clean). Workspace-wide clippy could not be run:vortex-duckdb's build script cannot download DuckDB sources in this sandbox.What APIs are changed? Are there any user-facing changes?
Schemehas a new required methodproduced_encodings() -> Vec<ArrayId>— a breaking change for third-party schemes, which must now declare their outputs.EditionSession::{enable, enabled, enabled_encodings},CascadingCompressor::with_enabled_editions_gating,BtrBlocksCompressorBuilder::{gate_by_enabled_editions, retain_allowed_encodings},FlatLayoutStrategy::with_enabled_editions_gating, andvortex::editions::DEFAULT_WRITE_EDITIONSare new.vortexfacade: encodings outside the enabled editions (fastlanes.delta,vortex.onpair,vortex.zstd_bufferswithout theunstable_encodingsfeature,vortex.patchedwithout the env opt-in) are no longer writable — writes containing them fail at the flat-layout validation instead of producing files without the read-forever guarantee. Sessions that enable no editions keep the previous staticALLOWED_ENCODINGSbehavior.🤖 Generated with Claude Code
https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2