diff --git a/Encryption.md b/Encryption.md index d3c8c9fa..733525c0 100644 --- a/Encryption.md +++ b/Encryption.md @@ -78,9 +78,9 @@ In a plaintext footer mode, the contents of the footer structure is visible and in order to verify its integrity. New footer fields keep an information about the file encryption algorithm and the footer signing key. -For encrypted columns, the following modules are always encrypted, with the same column key: -pages and page headers (both dictionary and data), column indexes, offset indexes, bloom filter -headers and bitsets. If the +For encrypted columns, the following modules are always encrypted, with the same column key: +pages and page headers (both dictionary and data), self-reference payloads, column indexes, +offset indexes, bloom filter headers and bitsets. If the column key is different from the footer encryption key, the column metadata is serialized separately and encrypted with the column key. In this case, the column metadata is also considered to be a module. @@ -135,11 +135,12 @@ related to the "uniqueness requirement of IVs and keys" (section 8 in the NIST s one IV is ever repeated, then the implementation may be vulnerable"*. *"Compliance with this requirement is crucial to the security of GCM"*. -The bulk of modules in a Parquet file are page headers and data pages. Therefore, one encryption -key shall not be used for more than 2^32 total module encryptions, as per the NIST specification. -Since each data page requires two module encryptions (header + data), this means in practice no -more than 2^31 pages per key. In Parquet files encrypted with multiple keys (footer and column -keys), the constraint on the number of invocations is applied to each key separately. +The bulk of modules in a Parquet file are page headers, data pages, and self-references. +Therefore, one encryption key shall not be used for more than 2^32 total module encryptions, +as per the NIST specification. Each data page requires two module encryptions (header and +data), and each self-reference requires one. In Parquet files encrypted with multiple keys +(footer and column keys), the constraint on the number of invocations is applied to each key +separately. When running in the context of a larger system, any particular Parquet writer implementation likely does not have sufficient context to enforce key invocation limits system-wide. Therefore, @@ -163,11 +164,11 @@ tag used to verify the ciphertext and AAD integrity. #### 4.2.2 AES_GCM_CTR_V1 -In this Parquet algorithm, all modules except pages are encrypted with the GCM cipher, as described -above. The pages are encrypted by the CTR cipher without padding. This makes it possible to encrypt/decrypt -the bulk of the data faster, while still verifying the metadata integrity and making -sure the file has not been replaced with a wrong version. However, tampering with the -page data might go unnoticed. The AES CTR cipher +In this Parquet algorithm, all modules except pages and self-references are encrypted with the +GCM cipher, as described above. Pages and self-references are encrypted by the CTR cipher +without padding. This makes it possible to encrypt/decrypt the bulk of the data faster, while +still verifying the metadata integrity and making sure the file has not been replaced with a +wrong version. However, tampering with page or self-reference data might go unnoticed. The AES CTR cipher must be implemented by a cryptographic provider according to the NIST SP 800-38A specification. In Parquet, an input to the CTR cipher is an encryption key, a 16-byte IV and a plaintext. IVs are comprised of @@ -256,7 +257,8 @@ Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenat 2. [All modules] module type (1 byte) 3. [All modules except footer] row group ordinal (2-byte short, little-endian) 4. [All modules except footer] column ordinal (2-byte short, little-endian) -5. [Data page and header only] page ordinal (2-byte short, little-endian) +5. [Data page and data page header only] page ordinal (2-byte short, little-endian) +6. [Self-reference only] self-reference ordinal within the column chunk (8-byte integer, little-endian) The following module types are defined: @@ -270,28 +272,35 @@ The following module types are defined: * OffsetIndex (7) * BloomFilter Header (8) * BloomFilter Bitset (9) + * Self-Reference (10) -| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal| -|----------------------|------------------|-------------|-------------------|----------------|-------------| -| Footer | yes | yes (0) | no | no | no | -| ColumnMetaData | yes | yes (1) | yes | yes | no | -| Data Page | yes | yes (2) | yes | yes | yes | -| Dictionary Page | yes | yes (3) | yes | yes | no | -| Data Page Header | yes | yes (4) | yes | yes | yes | -| Dictionary Page Header| yes | yes (5) | yes | yes | no | -| ColumnIndex | yes | yes (6) | yes | yes | no | -| OffsetIndex | yes | yes (7) | yes | yes | no | -| BloomFilter Header | yes | yes (8) | yes | yes | no | -| BloomFilter Bitset | yes | yes (9) | yes | yes | no | +| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Self-reference ordinal | +|-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------| +| Footer | yes | yes (0) | no | no | no | no | +| ColumnMetaData | yes | yes (1) | yes | yes | no | no | +| Data Page | yes | yes (2) | yes | yes | yes | no | +| Dictionary Page | yes | yes (3) | yes | yes | no | no | +| Data Page Header | yes | yes (4) | yes | yes | yes | no | +| Dictionary Page Header| yes | yes (5) | yes | yes | no | no | +| ColumnIndex | yes | yes (6) | yes | yes | no | no | +| OffsetIndex | yes | yes (7) | yes | yes | no | no | +| BloomFilter Header | yes | yes (8) | yes | yes | no | no | +| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | +| Self-Reference | yes | yes (10) | yes | yes | no | yes | + +For a self-reference, the column ordinal is that of the `inline` column. The +self-reference ordinal is the zero-based position representing the same `FILE` value +in that column chunk's repetition and definition level stream, including null +positions. It is derived by the reader and is not stored separately. ## 5. File Format ### 5.1 Encrypted module serialization -All modules, except column pages, are encrypted with the GCM cipher. In the AES_GCM_V1 algorithm, -the column pages are also encrypted with AES GCM. For each module, the GCM encryption +All modules, except column pages and self-references, are encrypted with the GCM cipher. In the +AES_GCM_V1 algorithm, column pages and self-references are also encrypted with AES GCM. For each module, the GCM encryption buffer is comprised of a nonce, ciphertext and tag, described in the Algorithms section. The length of the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself. @@ -299,14 +308,18 @@ the encryption buffer (a 4-byte little endian) is written to the output stream, |-----------------|------------------|------------------------------|----------------| -In the AES_GCM_CTR_V1 algorithm, the column pages are encrypted with AES CTR. -For each page, the CTR encryption buffer is comprised of a nonce and ciphertext, +In the AES_GCM_CTR_V1 algorithm, column pages and self-references are encrypted with AES CTR. +For each page or self-reference, the CTR encryption buffer is comprised of a nonce and ciphertext, described in the Algorithms section. The length of the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself. |length (4 bytes) | nonce (12 bytes) | ciphertext (length-12 bytes) | |-----------------|------------------|------------------------------| +For a self-reference, `offset` points to the beginning of the 4-byte length and `size` +is the size of the complete encrypted module, including the length, nonce, ciphertext, +and GCM tag when present. + ### 5.2 Crypto structures Parquet file encryption algorithm is specified in a union of the following Thrift structures: @@ -540,4 +553,3 @@ algorithm (no 16-byte tag in AES_GCM_CTR_V1) and with page configuration or data The throughput overhead of Parquet modular encryption depends on whether AES enciphering is done in software or hardware. In both cases, performing encryption on full pages (~1MB buffers) instead of on much smaller individual data values causes AES to work at its maximal speed. - diff --git a/LogicalTypes.md b/LogicalTypes.md index fae874c0..f807da1d 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -688,11 +688,11 @@ external reference (`uri` set). `offset` must not be < 0. ##### size -The byte length of the referenced data. Must be zero or a positive integer if set; a -value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set. -It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), -in which case the range runs to the end of the referenced file. Because a self-reference -always sets `offset`, it always sets `size` as well. +The byte length of the stored representation. Must be zero or a positive integer if set. +A value of 0 indicates an empty stored representation. `size` must be set whenever +`offset` is set. It may be omitted only for a whole-file external reference (`uri` set, +`offset` not set), in which case the range runs to the end of the referenced file. +Because a self-reference always sets `offset`, it always sets `size` as well. ##### content_type @@ -735,17 +735,17 @@ only. A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are set: -| `inline` | `uri` | `offset` | `size` | Resolves to | -|----------|-------|----------|--------|-------------------------------------------------------| -| set | - | - | - | the inline bytes | -| - | set | - | - | whole external file at `uri` | -| - | set | set | - | invalid | -| - | set | - | set | external `uri`, `[0, size)` | -| - | set | set | set | external `uri`, `[offset, offset + size)` | -| - | - | set | - | invalid | -| - | - | - | set | invalid | -| - | - | set | set | this file, `[offset, offset + size)` (self-reference) | -| - | - | - | - | nothing - invalid | +| `inline` | `uri` | `offset` | `size` | Resolves to | +|----------|-------|----------|--------|-----------------------------------------------------------------------| +| set | - | - | - | the inline bytes | +| - | set | - | - | whole external file at `uri` | +| - | set | set | - | invalid | +| - | set | - | set | external `uri`, `[0, size)` | +| - | set | set | set | external `uri`, `[offset, offset + size)` | +| - | - | set | - | invalid | +| - | - | - | set | invalid | +| - | - | set | set | stored bytes in this file, `[offset, offset + size)` (self-reference) | +| - | - | - | - | nothing - invalid | `size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. A self-reference (`uri` not set) must set `offset`, and therefore also @@ -756,10 +756,44 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +A schema that permits self-references must include the `inline` field. + +Each self-reference inherits the compression and encryption settings of the `inline` +column chunk in the same row group. The corresponding position is the position +representing the same `FILE` value in the `inline` column's repetition and definition +level stream. + +Each referenced byte range is compressed independently using the `CompressionCodec` +of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. + +Each compressed byte range is an independent compression block. Compression state is +not shared with the data page or with other referenced ranges. + +For an unencrypted self-reference, `offset` and `size` identify either the independent +compressed block or the uncompressed bytes. For a compressed block, the complete range +is supplied to the codec, and its decompressed output is the resolved value. + +The encryption state and key are inherited from the `inline` column chunk. If the +column chunk is encrypted, each self-reference is encrypted independently using the +same column key and file encryption algorithm. Compression is applied before +encryption. If the column chunk is not encrypted, its self-references are not +encrypted. See [Parquet Modular Encryption](Encryption.md) for the encryption layout +and AAD construction. + +A self-reference identifies a stored representation, not necessarily the resolved +bytes. Consumers must use a Parquet reader to resolve a self-reference; copying +`[offset, offset + size)` directly may return compressed or encrypted data. To resolve +a self-reference, a reader: + +1. reads the stored representation identified by `offset` and `size`; +2. decrypts it when the corresponding `inline` column chunk is encrypted; +3. decompresses it using the `CompressionCodec` of the corresponding `inline` column + chunk, unless the codec is `UNCOMPRESSED`; +4. returns the resulting bytes. + +`content_type` and `checksum` describe the resolved bytes after these transformations. +These compression and encryption rules do not apply to external references. Encryption +of external files referenced by `uri` is outside the scope of the Parquet format. #### Validation