Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 29 additions & 14 deletions LogicalTypes.md
Original file line number Diff line number Diff line change
Expand Up @@ -351,11 +351,18 @@ as shown below.

### TIMESTAMP

In data annotated with the `TIMESTAMP` logical type, each value is a single
`int64` number that can be decoded into year, month, day, hour, minute, second
and subsecond fields using calculations detailed below. Please note that a value
defined this way does not necessarily correspond to a single instant on the
time-line and such interpretations are allowed on purpose.
In data annotated with the `TIMESTAMP` logical type, each value is an `int64`
or a 12-byte `FIXED_LEN_BYTE_ARRAY` that can be decoded into year, month, day,
hour, minute, second and subsecond fields using calculations detailed below.
Please note that a value defined this way does not necessarily correspond to a
single instant on the time-line and such interpretations are allowed on purpose.

For the `int64` carrier, the value is a signed 64-bit integer count of `unit`s
since the Unix epoch.

For the `FIXED_LEN_BYTE_ARRAY` carrier (with `type_length = 12`), the value is a
signed 96-bit two's-complement little-endian integer count of `unit`s since the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest we do big-endian here so that we can use the same signed two complements byte compare we use for DECIMAL in FLBA.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose little-endian because it seems to match the rest of the spec (DECIMAL is the only deviation). But I don't have a strong preference, happy to change it to big-endian. I'd like to hear from others to see if anyone else agrees/disagrees.

Unix epoch.

The `TIMESTAMP` type has two type parameters:
- `isAdjustedToUTC` must be either `true` or `false`.
Expand Down Expand Up @@ -442,12 +449,14 @@ local timestamp in reality.

#### Common considerations

Every possible `int64` number represents a valid timestamp, but depending on the
precision, the corresponding year may be outside of the practical everyday
limits and implementations may choose to only support a limited range.
Every possible `int64` and `FIXED_LEN_BYTE_ARRAY(12)` value represents a valid
timestamp, but depending on the precision, the corresponding year may be outside
of the practical everyday limits and implementations may choose to only support
a limited range.

On the other hand, not every combination of year, month, day, hour, minute,
second and subsecond values can be encoded into an `int64`. Most notably:
second and subsecond values can be encoded into an `int64` or
`FIXED_LEN_BYTE_ARRAY(12)`. Most notably:

- An arbitrary combination of timestamp fields cannot be encoded as a single
number if the values for some of the fields are outside of their normal range
Expand All @@ -458,13 +467,19 @@ second and subsecond values can be encoded into an `int64`. Most notably:
- minute = 61
- month = 13
- day = 29, month = 2, year = any non-leap year
- Due to the range of the `int64` type, timestamps using the `NANOS` unit
- Due to the range of the `int64` type, `int64` timestamps using the `NANOS` unit
can only represent values between 1677-09-21 00:12:43 and 2262-04-11 23:47:16.
Values outside of this range cannot be represented with the `NANOS`
unit. (Other precisions have similar limits but those are outside of the
domain for practical everyday usage.)
Values outside of this range should instead be represented with the
`FIXED_LEN_BYTE_ARRAY(12)` physical type using the `NANOS` unit. Other precisions
have similar limits but those are outside of the domain for practical everyday use.

The sort order used for `TIMESTAMP` is signed:

The sort order used for `TIMESTAMP` is signed.
- For the `int64` carrier: signed integer comparison.
- For the `FIXED_LEN_BYTE_ARRAY(12)` carrier: signed two's-complement comparison of
the represented value. The correct ordering can be produced by flipping the
most-significant bit of the last byte and then using unsigned byte-wise comparison
starting from the last byte.

#### Deprecated timestamp ConvertedType

Expand Down
2 changes: 1 addition & 1 deletion src/main/thrift/parquet.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -365,7 +365,7 @@ union TimeUnit {
/**
* Timestamp logical type annotation
*
* Allowed for physical types: INT64
* Allowed for physical types: INT64 and FIXED_LEN_BYTE_ARRAY with `type_length` = 12.
*/
struct TimestampType {
1: required bool isAdjustedToUTC
Expand Down