From 53dc52bf86d41869bb0c5ed691af669dd29ec129 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Wed, 29 Jul 2026 20:02:41 +0300 Subject: [PATCH 1/6] Specify compression for FILE self-references --- LogicalTypes.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/LogicalTypes.md b/LogicalTypes.md index fae874c0..7fb4f079 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -756,6 +756,10 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. +The bytes referenced by a self-reference are compressed with the `CompressionCodec` +of the inline column chunk's `ColumnMetaData`. This does not apply to external +references. + Parquet files containing self-references must not use Parquet modular encryption. Self-referenced byte ranges are not Parquet encryption modules and therefore cannot be encrypted or authenticated independently. Encryption of external files referenced From 7b729c6a42766790fcf780000d338a710d9aabbd Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 30 Jul 2026 02:30:48 +0300 Subject: [PATCH 2/6] Define FILE self-reference compression semantics --- LogicalTypes.md | 37 +++++++++++++++++++++++++++++-------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 7fb4f079..006dfd9c 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -688,11 +688,11 @@ external reference (`uri` set). `offset` must not be < 0. ##### size -The byte length of the referenced data. Must be zero or a positive integer if set; a -value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set. -It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), -in which case the range runs to the end of the referenced file. Because a self-reference -always sets `offset`, it always sets `size` as well. +The byte length of the stored representation. Must be zero or a positive integer if set. +A value of 0 indicates an empty stored representation. `size` must be set whenever +`offset` is set. It may be omitted only for a whole-file external reference (`uri` set, +`offset` not set), in which case the range runs to the end of the referenced file. +Because a self-reference always sets `offset`, it always sets `size` as well. ##### content_type @@ -756,9 +756,30 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -The bytes referenced by a self-reference are compressed with the `CompressionCodec` -of the inline column chunk's `ColumnMetaData`. This does not apply to external -references. +A schema that permits self-references must include the `inline` field and encode it +using Data Page V2. + +Each self-reference corresponds positionally to a value in the `inline` column. Its +compression state is inherited from the Data Page V2 containing that position: + +* If `is_compressed` is true, the referenced byte range is compressed independently + using the `CompressionCodec` of the `inline` column chunk. +* If `is_compressed` is false, the referenced byte range contains the resolved bytes + without compression. + +All self-references corresponding to positions in the same data page share the page's +compression decision. A writer that requires a different decision must begin a new +Data Page V2. + +Each compressed byte range is an independent compression block. Compression state is +not shared with the data page or with other referenced ranges. + +For a self-reference, `offset` and `size` identify the stored representation. When +compressed, `size` is the compressed byte length. The complete range is supplied to +the codec, and its decompressed output is the resolved value. + +`content_type` and `checksum` describe the resolved bytes after decompression. These +compression rules do not apply to external references. Parquet files containing self-references must not use Parquet modular encryption. Self-referenced byte ranges are not Parquet encryption modules and therefore cannot From f5eb8854ca10b9b990a197424e5b4a40d6654db5 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 30 Jul 2026 09:34:48 +0300 Subject: [PATCH 3/6] Define inherited encryption for FILE self-references --- Encryption.md | 74 ++++++++++++++++++++++++++++--------------------- LogicalTypes.md | 44 +++++++++++++++++++---------- 2 files changed, 73 insertions(+), 45 deletions(-) diff --git a/Encryption.md b/Encryption.md index d3c8c9fa..9625772a 100644 --- a/Encryption.md +++ b/Encryption.md @@ -78,9 +78,9 @@ In a plaintext footer mode, the contents of the footer structure is visible and in order to verify its integrity. New footer fields keep an information about the file encryption algorithm and the footer signing key. -For encrypted columns, the following modules are always encrypted, with the same column key: -pages and page headers (both dictionary and data), column indexes, offset indexes, bloom filter -headers and bitsets. If the +For encrypted columns, the following modules are always encrypted, with the same column key: +pages and page headers (both dictionary and data), self-reference payloads, column indexes, +offset indexes, bloom filter headers and bitsets. If the column key is different from the footer encryption key, the column metadata is serialized separately and encrypted with the column key. In this case, the column metadata is also considered to be a module. @@ -135,11 +135,12 @@ related to the "uniqueness requirement of IVs and keys" (section 8 in the NIST s one IV is ever repeated, then the implementation may be vulnerable"*. *"Compliance with this requirement is crucial to the security of GCM"*. -The bulk of modules in a Parquet file are page headers and data pages. Therefore, one encryption -key shall not be used for more than 2^32 total module encryptions, as per the NIST specification. -Since each data page requires two module encryptions (header + data), this means in practice no -more than 2^31 pages per key. In Parquet files encrypted with multiple keys (footer and column -keys), the constraint on the number of invocations is applied to each key separately. +The bulk of modules in a Parquet file are page headers, data pages, and self-references. +Therefore, one encryption key shall not be used for more than 2^32 total module encryptions, +as per the NIST specification. Each data page requires two module encryptions (header and +data), and each self-reference requires one. In Parquet files encrypted with multiple keys +(footer and column keys), the constraint on the number of invocations is applied to each key +separately. When running in the context of a larger system, any particular Parquet writer implementation likely does not have sufficient context to enforce key invocation limits system-wide. Therefore, @@ -163,11 +164,11 @@ tag used to verify the ciphertext and AAD integrity. #### 4.2.2 AES_GCM_CTR_V1 -In this Parquet algorithm, all modules except pages are encrypted with the GCM cipher, as described -above. The pages are encrypted by the CTR cipher without padding. This makes it possible to encrypt/decrypt -the bulk of the data faster, while still verifying the metadata integrity and making -sure the file has not been replaced with a wrong version. However, tampering with the -page data might go unnoticed. The AES CTR cipher +In this Parquet algorithm, all modules except pages and self-references are encrypted with the +GCM cipher, as described above. Pages and self-references are encrypted by the CTR cipher +without padding. This makes it possible to encrypt/decrypt the bulk of the data faster, while +still verifying the metadata integrity and making sure the file has not been replaced with a +wrong version. However, tampering with page or self-reference data might go unnoticed. The AES CTR cipher must be implemented by a cryptographic provider according to the NIST SP 800-38A specification. In Parquet, an input to the CTR cipher is an encryption key, a 16-byte IV and a plaintext. IVs are comprised of @@ -256,7 +257,8 @@ Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenat 2. [All modules] module type (1 byte) 3. [All modules except footer] row group ordinal (2-byte short, little-endian) 4. [All modules except footer] column ordinal (2-byte short, little-endian) -5. [Data page and header only] page ordinal (2-byte short, little-endian) +5. [Data page, data page header, and self-reference only] page ordinal (2-byte short, little-endian) +6. [Self-reference only] value ordinal within the data page (4-byte integer, little-endian) The following module types are defined: @@ -270,28 +272,35 @@ The following module types are defined: * OffsetIndex (7) * BloomFilter Header (8) * BloomFilter Bitset (9) + * Self-Reference (10) -| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal| -|----------------------|------------------|-------------|-------------------|----------------|-------------| -| Footer | yes | yes (0) | no | no | no | -| ColumnMetaData | yes | yes (1) | yes | yes | no | -| Data Page | yes | yes (2) | yes | yes | yes | -| Dictionary Page | yes | yes (3) | yes | yes | no | -| Data Page Header | yes | yes (4) | yes | yes | yes | -| Dictionary Page Header| yes | yes (5) | yes | yes | no | -| ColumnIndex | yes | yes (6) | yes | yes | no | -| OffsetIndex | yes | yes (7) | yes | yes | no | -| BloomFilter Header | yes | yes (8) | yes | yes | no | -| BloomFilter Bitset | yes | yes (9) | yes | yes | no | +| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Value ordinal | +|-----------------------|------------------|-------------|-------------------|----------------|--------------|---------------| +| Footer | yes | yes (0) | no | no | no | no | +| ColumnMetaData | yes | yes (1) | yes | yes | no | no | +| Data Page | yes | yes (2) | yes | yes | yes | no | +| Dictionary Page | yes | yes (3) | yes | yes | no | no | +| Data Page Header | yes | yes (4) | yes | yes | yes | no | +| Dictionary Page Header| yes | yes (5) | yes | yes | no | no | +| ColumnIndex | yes | yes (6) | yes | yes | no | no | +| OffsetIndex | yes | yes (7) | yes | yes | no | no | +| BloomFilter Header | yes | yes (8) | yes | yes | no | no | +| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | +| Self-Reference | yes | yes (10) | yes | yes | yes | yes | + +For a self-reference, the column ordinal and page ordinal are those of the corresponding +position in the `inline` column. The value ordinal is the zero-based position within that +page, counting every position represented by its repetition and definition levels, +including null positions. ## 5. File Format ### 5.1 Encrypted module serialization -All modules, except column pages, are encrypted with the GCM cipher. In the AES_GCM_V1 algorithm, -the column pages are also encrypted with AES GCM. For each module, the GCM encryption +All modules, except column pages and self-references, are encrypted with the GCM cipher. In the +AES_GCM_V1 algorithm, column pages and self-references are also encrypted with AES GCM. For each module, the GCM encryption buffer is comprised of a nonce, ciphertext and tag, described in the Algorithms section. The length of the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself. @@ -299,14 +308,18 @@ the encryption buffer (a 4-byte little endian) is written to the output stream, |-----------------|------------------|------------------------------|----------------| -In the AES_GCM_CTR_V1 algorithm, the column pages are encrypted with AES CTR. -For each page, the CTR encryption buffer is comprised of a nonce and ciphertext, +In the AES_GCM_CTR_V1 algorithm, column pages and self-references are encrypted with AES CTR. +For each page or self-reference, the CTR encryption buffer is comprised of a nonce and ciphertext, described in the Algorithms section. The length of the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself. |length (4 bytes) | nonce (12 bytes) | ciphertext (length-12 bytes) | |-----------------|------------------|------------------------------| +For a self-reference, `offset` points to the beginning of the 4-byte length and `size` +is the size of the complete encrypted module, including the length, nonce, ciphertext, +and GCM tag when present. + ### 5.2 Crypto structures Parquet file encryption algorithm is specified in a union of the following Thrift structures: @@ -540,4 +553,3 @@ algorithm (no 16-byte tag in AES_GCM_CTR_V1) and with page configuration or data The throughput overhead of Parquet modular encryption depends on whether AES enciphering is done in software or hardware. In both cases, performing encryption on full pages (~1MB buffers) instead of on much smaller individual data values causes AES to work at its maximal speed. - diff --git a/LogicalTypes.md b/LogicalTypes.md index 006dfd9c..da7a9b67 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -744,7 +744,7 @@ set: | - | set | set | set | external `uri`, `[offset, offset + size)` | | - | - | set | - | invalid | | - | - | - | set | invalid | -| - | - | set | set | this file, `[offset, offset + size)` (self-reference) | +| - | - | set | set | stored bytes in this file, `[offset, offset + size)` (self-reference) | | - | - | - | - | nothing - invalid | `size` must be set whenever `offset` is set, so any offset-based read always carries an @@ -759,8 +759,11 @@ can be renamed or relocated as a single unit. A schema that permits self-references must include the `inline` field and encode it using Data Page V2. -Each self-reference corresponds positionally to a value in the `inline` column. Its -compression state is inherited from the Data Page V2 containing that position: +Each self-reference inherits the compression and encryption settings of the `inline` +column at the same position. The corresponding position is the position representing +the same `FILE` value in the `inline` column's repetition and definition level stream. + +The compression state is inherited from the Data Page V2 containing that position: * If `is_compressed` is true, the referenced byte range is compressed independently using the `CompressionCodec` of the `inline` column chunk. @@ -774,17 +777,30 @@ Data Page V2. Each compressed byte range is an independent compression block. Compression state is not shared with the data page or with other referenced ranges. -For a self-reference, `offset` and `size` identify the stored representation. When -compressed, `size` is the compressed byte length. The complete range is supplied to -the codec, and its decompressed output is the resolved value. - -`content_type` and `checksum` describe the resolved bytes after decompression. These -compression rules do not apply to external references. - -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +For an unencrypted self-reference, `offset` and `size` identify either the independent +compressed block or the uncompressed bytes. For a compressed block, the complete range +is supplied to the codec, and its decompressed output is the resolved value. + +The encryption state and key are inherited from the `inline` column chunk. If the +column chunk is encrypted, each self-reference is encrypted independently using the +same column key and file encryption algorithm. Compression is applied before +encryption. If the column chunk is not encrypted, its self-references are not +encrypted. See [Parquet Modular Encryption](Encryption.md) for the encryption layout +and AAD construction. + +A self-reference identifies a stored representation, not necessarily the resolved +bytes. Consumers must use a Parquet reader to resolve a self-reference; copying +`[offset, offset + size)` directly may return compressed or encrypted data. To resolve +a self-reference, a reader: + +1. reads the stored representation identified by `offset` and `size`; +2. decrypts it when the corresponding `inline` column chunk is encrypted; +3. decompresses it when `is_compressed` is true for the corresponding Data Page V2; +4. returns the resulting bytes. + +`content_type` and `checksum` describe the resolved bytes after these transformations. +These compression and encryption rules do not apply to external references. Encryption +of external files referenced by `uri` is outside the scope of the Parquet format. #### Validation From 86479df673a307e69eac9cc1f6318747cff4001a Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 30 Jul 2026 09:42:55 +0300 Subject: [PATCH 4/6] Avoid prescribing FILE page boundaries --- LogicalTypes.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index da7a9b67..257e3a95 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -770,10 +770,6 @@ The compression state is inherited from the Data Page V2 containing that positio * If `is_compressed` is false, the referenced byte range contains the resolved bytes without compression. -All self-references corresponding to positions in the same data page share the page's -compression decision. A writer that requires a different decision must begin a new -Data Page V2. - Each compressed byte range is an independent compression block. Compression state is not shared with the data page or with other referenced ranges. From 3047a52fb64b0f0cde35fb2b4474f632064761eb Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 30 Jul 2026 19:14:37 +0300 Subject: [PATCH 5/6] Make FILE storage inheritance page-independent --- Encryption.md | 40 ++++++++++++++++++++-------------------- LogicalTypes.md | 19 ++++++++----------- 2 files changed, 28 insertions(+), 31 deletions(-) diff --git a/Encryption.md b/Encryption.md index 9625772a..733525c0 100644 --- a/Encryption.md +++ b/Encryption.md @@ -257,8 +257,8 @@ Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenat 2. [All modules] module type (1 byte) 3. [All modules except footer] row group ordinal (2-byte short, little-endian) 4. [All modules except footer] column ordinal (2-byte short, little-endian) -5. [Data page, data page header, and self-reference only] page ordinal (2-byte short, little-endian) -6. [Self-reference only] value ordinal within the data page (4-byte integer, little-endian) +5. [Data page and data page header only] page ordinal (2-byte short, little-endian) +6. [Self-reference only] self-reference ordinal within the column chunk (8-byte integer, little-endian) The following module types are defined: @@ -275,24 +275,24 @@ The following module types are defined: * Self-Reference (10) -| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Value ordinal | -|-----------------------|------------------|-------------|-------------------|----------------|--------------|---------------| -| Footer | yes | yes (0) | no | no | no | no | -| ColumnMetaData | yes | yes (1) | yes | yes | no | no | -| Data Page | yes | yes (2) | yes | yes | yes | no | -| Dictionary Page | yes | yes (3) | yes | yes | no | no | -| Data Page Header | yes | yes (4) | yes | yes | yes | no | -| Dictionary Page Header| yes | yes (5) | yes | yes | no | no | -| ColumnIndex | yes | yes (6) | yes | yes | no | no | -| OffsetIndex | yes | yes (7) | yes | yes | no | no | -| BloomFilter Header | yes | yes (8) | yes | yes | no | no | -| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | -| Self-Reference | yes | yes (10) | yes | yes | yes | yes | - -For a self-reference, the column ordinal and page ordinal are those of the corresponding -position in the `inline` column. The value ordinal is the zero-based position within that -page, counting every position represented by its repetition and definition levels, -including null positions. +| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Self-reference ordinal | +|-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------| +| Footer | yes | yes (0) | no | no | no | no | +| ColumnMetaData | yes | yes (1) | yes | yes | no | no | +| Data Page | yes | yes (2) | yes | yes | yes | no | +| Dictionary Page | yes | yes (3) | yes | yes | no | no | +| Data Page Header | yes | yes (4) | yes | yes | yes | no | +| Dictionary Page Header| yes | yes (5) | yes | yes | no | no | +| ColumnIndex | yes | yes (6) | yes | yes | no | no | +| OffsetIndex | yes | yes (7) | yes | yes | no | no | +| BloomFilter Header | yes | yes (8) | yes | yes | no | no | +| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | +| Self-Reference | yes | yes (10) | yes | yes | no | yes | + +For a self-reference, the column ordinal is that of the `inline` column. The +self-reference ordinal is the zero-based position representing the same `FILE` value +in that column chunk's repetition and definition level stream, including null +positions. It is derived by the reader and is not stored separately. diff --git a/LogicalTypes.md b/LogicalTypes.md index 257e3a95..e90b407e 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -756,19 +756,15 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -A schema that permits self-references must include the `inline` field and encode it -using Data Page V2. +A schema that permits self-references must include the `inline` field. Each self-reference inherits the compression and encryption settings of the `inline` -column at the same position. The corresponding position is the position representing -the same `FILE` value in the `inline` column's repetition and definition level stream. +column chunk in the same row group. The corresponding position is the position +representing the same `FILE` value in the `inline` column's repetition and definition +level stream. -The compression state is inherited from the Data Page V2 containing that position: - -* If `is_compressed` is true, the referenced byte range is compressed independently - using the `CompressionCodec` of the `inline` column chunk. -* If `is_compressed` is false, the referenced byte range contains the resolved bytes - without compression. +Each referenced byte range is compressed independently using the `CompressionCodec` +of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. Each compressed byte range is an independent compression block. Compression state is not shared with the data page or with other referenced ranges. @@ -791,7 +787,8 @@ a self-reference, a reader: 1. reads the stored representation identified by `offset` and `size`; 2. decrypts it when the corresponding `inline` column chunk is encrypted; -3. decompresses it when `is_compressed` is true for the corresponding Data Page V2; +3. decompresses it using the `CompressionCodec` of the corresponding `inline` column + chunk, unless the codec is `UNCOMPRESSED`; 4. returns the resulting bytes. `content_type` and `checksum` describe the resolved bytes after these transformations. From 81f61ed74074d58794953ea330e64aee6ae879a1 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Fri, 31 Jul 2026 00:35:08 +0300 Subject: [PATCH 6/6] Align FILE resolution table --- LogicalTypes.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index e90b407e..f807da1d 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -735,17 +735,17 @@ only. A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are set: -| `inline` | `uri` | `offset` | `size` | Resolves to | -|----------|-------|----------|--------|-------------------------------------------------------| -| set | - | - | - | the inline bytes | -| - | set | - | - | whole external file at `uri` | -| - | set | set | - | invalid | -| - | set | - | set | external `uri`, `[0, size)` | -| - | set | set | set | external `uri`, `[offset, offset + size)` | -| - | - | set | - | invalid | -| - | - | - | set | invalid | +| `inline` | `uri` | `offset` | `size` | Resolves to | +|----------|-------|----------|--------|-----------------------------------------------------------------------| +| set | - | - | - | the inline bytes | +| - | set | - | - | whole external file at `uri` | +| - | set | set | - | invalid | +| - | set | - | set | external `uri`, `[0, size)` | +| - | set | set | set | external `uri`, `[offset, offset + size)` | +| - | - | set | - | invalid | +| - | - | - | set | invalid | | - | - | set | set | stored bytes in this file, `[offset, offset + size)` (self-reference) | -| - | - | - | - | nothing - invalid | +| - | - | - | - | nothing - invalid | `size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. A self-reference (`uri` not set) must set `offset`, and therefore also