Skip to content

Specify storage inheritance for FILE self-references #603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alkis wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Specify storage inheritance for FILE self-references #603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
53dc52b
Specify compression for FILE self-references
alkis Jul 29, 2026
7b729c6
Define FILE self-reference compression semantics
alkis Jul 29, 2026
f5eb885
Define inherited encryption for FILE self-references
alkis Jul 30, 2026
86479df
Avoid prescribing FILE page boundaries
alkis Jul 30, 2026
3047a52
Make FILE storage inheritance page-independent
alkis Jul 30, 2026
81f61ed
Align FILE resolution table
alkis Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 43 additions & 31 deletions Encryption.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,9 @@ In a plaintext footer mode, the contents of the footer structure is visible and
in order to verify its integrity. New footer fields keep an
information about the file encryption algorithm and the footer signing key.

For encrypted columns, the following modules are always encrypted, with the same column key:
pages and page headers (both dictionary and data), column indexes, offset indexes, bloom filter
headers and bitsets. If the
For encrypted columns, the following modules are always encrypted, with the same column key:
pages and page headers (both dictionary and data), self-reference payloads, column indexes,
offset indexes, bloom filter headers and bitsets. If the
column key is different from the footer encryption key, the column metadata is serialized
separately and encrypted with the column key. In this case, the column metadata is also
considered to be a module.
Expand Down Expand Up @@ -135,11 +135,12 @@ related to the "uniqueness requirement of IVs and keys" (section 8 in the NIST s
one IV is ever repeated, then the implementation may be vulnerable"*. *"Compliance with this
requirement is crucial to the security of GCM"*.

The bulk of modules in a Parquet file are page headers and data pages. Therefore, one encryption
key shall not be used for more than 2^32 total module encryptions, as per the NIST specification.
Since each data page requires two module encryptions (header + data), this means in practice no
more than 2^31 pages per key. In Parquet files encrypted with multiple keys (footer and column
keys), the constraint on the number of invocations is applied to each key separately.
The bulk of modules in a Parquet file are page headers, data pages, and self-references.
Therefore, one encryption key shall not be used for more than 2^32 total module encryptions,
as per the NIST specification. Each data page requires two module encryptions (header and
data), and each self-reference requires one. In Parquet files encrypted with multiple keys
(footer and column keys), the constraint on the number of invocations is applied to each key
separately.

When running in the context of a larger system, any particular Parquet writer implementation likely
does not have sufficient context to enforce key invocation limits system-wide. Therefore,
Expand All @@ -163,11 +164,11 @@ tag used to verify the ciphertext and AAD integrity.

#### 4.2.2 AES_GCM_CTR_V1

In this Parquet algorithm, all modules except pages are encrypted with the GCM cipher, as described
above. The pages are encrypted by the CTR cipher without padding. This makes it possible to encrypt/decrypt
the bulk of the data faster, while still verifying the metadata integrity and making
sure the file has not been replaced with a wrong version. However, tampering with the
page data might go unnoticed. The AES CTR cipher
In this Parquet algorithm, all modules except pages and self-references are encrypted with the
GCM cipher, as described above. Pages and self-references are encrypted by the CTR cipher
without padding. This makes it possible to encrypt/decrypt the bulk of the data faster, while
still verifying the metadata integrity and making sure the file has not been replaced with a
wrong version. However, tampering with page or self-reference data might go unnoticed. The AES CTR cipher
must be implemented by a cryptographic provider according to the NIST SP 800-38A specification.

In Parquet, an input to the CTR cipher is an encryption key, a 16-byte IV and a plaintext. IVs are comprised of
Expand Down Expand Up @@ -256,7 +257,8 @@ Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenat
2. [All modules] module type (1 byte)
3. [All modules except footer] row group ordinal (2-byte short, little-endian)
4. [All modules except footer] column ordinal (2-byte short, little-endian)
5. [Data page and header only] page ordinal (2-byte short, little-endian)
5. [Data page and data page header only] page ordinal (2-byte short, little-endian)
6. [Self-reference only] self-reference ordinal within the column chunk (8-byte integer, little-endian)

The following module types are defined:

Expand All @@ -270,43 +272,54 @@ The following module types are defined:
* OffsetIndex (7)
* BloomFilter Header (8)
* BloomFilter Bitset (9)
* Self-Reference (10)


| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal|
|----------------------|------------------|-------------|-------------------|----------------|-------------|
| Footer | yes | yes (0) | no | no | no |
| ColumnMetaData | yes | yes (1) | yes | yes | no |
| Data Page | yes | yes (2) | yes | yes | yes |
| Dictionary Page | yes | yes (3) | yes | yes | no |
| Data Page Header | yes | yes (4) | yes | yes | yes |
| Dictionary Page Header| yes | yes (5) | yes | yes | no |
| ColumnIndex | yes | yes (6) | yes | yes | no |
| OffsetIndex | yes | yes (7) | yes | yes | no |
| BloomFilter Header | yes | yes (8) | yes | yes | no |
| BloomFilter Bitset | yes | yes (9) | yes | yes | no |
| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Self-reference ordinal |
|-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------|
| Footer | yes | yes (0) | no | no | no | no |
| ColumnMetaData | yes | yes (1) | yes | yes | no | no |
| Data Page | yes | yes (2) | yes | yes | yes | no |
| Dictionary Page | yes | yes (3) | yes | yes | no | no |
| Data Page Header | yes | yes (4) | yes | yes | yes | no |
| Dictionary Page Header| yes | yes (5) | yes | yes | no | no |
| ColumnIndex | yes | yes (6) | yes | yes | no | no |
| OffsetIndex | yes | yes (7) | yes | yes | no | no |
| BloomFilter Header | yes | yes (8) | yes | yes | no | no |
| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no |
| Self-Reference | yes | yes (10) | yes | yes | no | yes |

For a self-reference, the column ordinal is that of the `inline` column. The
self-reference ordinal is the zero-based position representing the same `FILE` value
in that column chunk's repetition and definition level stream, including null
positions. It is derived by the reader and is not stored separately.



## 5. File Format

### 5.1 Encrypted module serialization
All modules, except column pages, are encrypted with the GCM cipher. In the AES_GCM_V1 algorithm,
the column pages are also encrypted with AES GCM. For each module, the GCM encryption
All modules, except column pages and self-references, are encrypted with the GCM cipher. In the
AES_GCM_V1 algorithm, column pages and self-references are also encrypted with AES GCM. For each module, the GCM encryption
buffer is comprised of a nonce, ciphertext and tag, described in the Algorithms section. The length of
the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself.

|length (4 bytes) | nonce (12 bytes) | ciphertext (length-28 bytes) | tag (16 bytes) |
|-----------------|------------------|------------------------------|----------------|


In the AES_GCM_CTR_V1 algorithm, the column pages are encrypted with AES CTR.
For each page, the CTR encryption buffer is comprised of a nonce and ciphertext,
In the AES_GCM_CTR_V1 algorithm, column pages and self-references are encrypted with AES CTR.
For each page or self-reference, the CTR encryption buffer is comprised of a nonce and ciphertext,
described in the Algorithms section. The length of the encryption buffer
(a 4-byte little endian) is written to the output stream, followed by the buffer itself.

|length (4 bytes) | nonce (12 bytes) | ciphertext (length-12 bytes) |
|-----------------|------------------|------------------------------|

For a self-reference, `offset` points to the beginning of the 4-byte length and `size`
is the size of the complete encrypted module, including the length, nonce, ciphertext,
and GCM tag when present.
Comment on lines +319 to +321

@rok rok Jul 31, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4-byte length limits us to 2GiB payload. Are we expecting use cases with greater size requirements?



### 5.2 Crypto structures
Parquet file encryption algorithm is specified in a union of the following Thrift structures:
Expand Down Expand Up @@ -540,4 +553,3 @@ algorithm (no 16-byte tag in AES_GCM_CTR_V1) and with page configuration or data
The throughput overhead of Parquet modular encryption depends on whether AES enciphering is
done in software or hardware. In both cases, performing encryption on full pages (~1MB buffers)
instead of on much smaller individual data values causes AES to work at its maximal speed.

74 changes: 54 additions & 20 deletions LogicalTypes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -688,11 +688,11 @@ external reference (`uri` set). `offset` must not be < 0.

##### size

The byte length of the referenced data. Must be zero or a positive integer if set; a
value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
It may be omitted only for a whole-file external reference (`uri` set, `offset` not set),
in which case the range runs to the end of the referenced file. Because a self-reference
always sets `offset`, it always sets `size` as well.
The byte length of the stored representation. Must be zero or a positive integer if set.
A value of 0 indicates an empty stored representation. `size` must be set whenever
`offset` is set. It may be omitted only for a whole-file external reference (`uri` set,
`offset` not set), in which case the range runs to the end of the referenced file.
Because a self-reference always sets `offset`, it always sets `size` as well.

##### content_type

Expand Down Expand Up @@ -735,17 +735,17 @@ only.
A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are
set:

| `inline` | `uri` | `offset` | `size` | Resolves to |
|----------|-------|----------|--------|-------------------------------------------------------|
| set | - | - | - | the inline bytes |
| - | set | - | - | whole external file at `uri` |
| - | set | set | - | invalid |
| - | set | - | set | external `uri`, `[0, size)` |
| - | set | set | set | external `uri`, `[offset, offset + size)` |
| - | - | set | - | invalid |
| - | - | - | set | invalid |
| - | - | set | set | this file, `[offset, offset + size)` (self-reference) |

@brkyvz brkyvz Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mind updating the spaces for the other lines so that the table is aligned?

@alkis alkis Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed—the table is aligned now. Thanks!

| - | - | - | - | nothing - invalid |
| `inline` | `uri` | `offset` | `size` | Resolves to |
|----------|-------|----------|--------|-----------------------------------------------------------------------|
| set | - | - | - | the inline bytes |
| - | set | - | - | whole external file at `uri` |
| - | set | set | - | invalid |
| - | set | - | set | external `uri`, `[0, size)` |
| - | set | set | set | external `uri`, `[offset, offset + size)` |
| - | - | set | - | invalid |
| - | - | - | set | invalid |
| - | - | set | set | stored bytes in this file, `[offset, offset + size)` (self-reference) |
| - | - | - | - | nothing - invalid |

`size` must be set whenever `offset` is set, so any offset-based read always carries an
explicit `size`. A self-reference (`uri` not set) must set `offset`, and therefore also
Expand All @@ -756,10 +756,44 @@ A self-reference points within the same Parquet file using `offset` and `size` (
required). A self-reference is when `uri` is not set. A file containing self-references
can be renamed or relocated as a single unit.

Parquet files containing self-references must not use Parquet modular encryption.
Self-referenced byte ranges are not Parquet encryption modules and therefore cannot
be encrypted or authenticated independently. Encryption of external files referenced
by `uri` is outside the scope of the Parquet format.
A schema that permits self-references must include the `inline` field.

Each self-reference inherits the compression and encryption settings of the `inline`
column chunk in the same row group. The corresponding position is the position
representing the same `FILE` value in the `inline` column's repetition and definition
level stream.

Each referenced byte range is compressed independently using the `CompressionCodec`
of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed.

Each compressed byte range is an independent compression block. Compression state is
not shared with the data page or with other referenced ranges.

For an unencrypted self-reference, `offset` and `size` identify either the independent
compressed block or the uncompressed bytes. For a compressed block, the complete range
is supplied to the codec, and its decompressed output is the resolved value.
Comment on lines +773 to +774

@rok rok Jul 31, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also store the decompressed size somewhere?


The encryption state and key are inherited from the `inline` column chunk. If the
column chunk is encrypted, each self-reference is encrypted independently using the
same column key and file encryption algorithm. Compression is applied before
encryption. If the column chunk is not encrypted, its self-references are not
encrypted. See [Parquet Modular Encryption](Encryption.md) for the encryption layout
and AAD construction.

A self-reference identifies a stored representation, not necessarily the resolved
bytes. Consumers must use a Parquet reader to resolve a self-reference; copying
`[offset, offset + size)` directly may return compressed or encrypted data. To resolve
a self-reference, a reader:

1. reads the stored representation identified by `offset` and `size`;
2. decrypts it when the corresponding `inline` column chunk is encrypted;
3. decompresses it using the `CompressionCodec` of the corresponding `inline` column
chunk, unless the codec is `UNCOMPRESSED`;
4. returns the resulting bytes.

`content_type` and `checksum` describe the resolved bytes after these transformations.
These compression and encryption rules do not apply to external references. Encryption
of external files referenced by `uri` is outside the scope of the Parquet format.

#### Validation

Expand Down