[GH-583]: Introduce chronological ordering for INT96 timestamps

[divjotarora]

Rationale for this change

When writing INT96 timestamp columns, writers either omit stats altogether or emit stats using the TYPE_ORDER ordering. However, some writers were incorrectly emitting stats via bytewise comparisons, which does not result in chronological INT96 ordering. These stats are incorrect and must be ignored by readers. The goal of this change is to introduce a mechanism for readers to correctly determine the validity of an INT96 column's statistics and ignore them if they are potentially incorrect.

What changes are included in this PR?

This PR specifies a new INT96_TIMESTAMP_ORDER sort order specifically used for INT96 timestamp statistics. Additionally, it suggests writers use this ordering when emitting INT96 stats and that readers ignore stats for INT96 TYPE_ORDER'd columns.

Do these changes have PoC implementations?

In-progress

Closes #$583

[wgtmac]

I have some general questions with regard to this proposal. A new column order requires change on the writer side. If we need to change the writer code anyway, isn't it simpler to just let writers emit INT64 values for timestamp? If it is targeted to not break the old readers that currently consume INT96 values and unable to upgrade, we have to make sure that they do not fail because of the unknown column order.

[emkornfield]

The wording looks fine to me. I agree with @etseidl on the clarification. And also we should try to figure out if there is a way to address @wgtmac concerns.

[etseidl]

I don't think the intent here is to support old readers. Instead, this is to aid new readers that want to be able to trust INT96 statistics generated by new writers.

We've already established that most old readers should be able to tolerate an unknown column order, with arrow-rs pre-57.0 being a notable exception.

[wgtmac]

this is to aid new readers that want to be able to trust INT96 statistics generated by new writers

What is the point of this? If both readers and writers are new, why not using INT64-typed timestamps instead? INT96 are marked as deprecated for years.

[divjotarora]

What is the point of this? If both readers and writers are new, why not using INT64-typed timestamps instead? INT96 are marked as deprecated for years.

@wgtmac This is a fair point, but some large engines like Spark still default to writing INT96 timestamps and not having a way to signal that stats are definitively correct is useful there. I'm also following up with Spark folks to see how we can make progress and move towards INT64 timestamps by default, but in the short term I still feel this change is targeted and useful.

If it is targeted to not break the old readers that currently consume INT96 values and unable to upgrade, we have to make sure that they do not fail because of the unknown column order.

This change is no more dangerous / breaking for old readers than the recent change to add the IEEE floating point order. IIRC there was a message in the mailing list about some older versions of arrow-rs failing on unknown sort orders, but I believe that's been fixed as well.

[etseidl]

Thanks, changes look good 👍

BTW I have a rust PoC nearly ready to go.

[etseidl]

Still needs to add tests, but Rust PoC is up apache/arrow-rs#10106

[emkornfield]

What is the point of this? If both readers and writers are new, why not using INT64-typed timestamps instead? INT96 are marked as deprecated for years.

@wgtmac This is a fair point, but some large engines like Spark still default to writing INT96 timestamps and not having a way to signal that stats are definitively correct is useful there. I'm also following up with Spark folks to see how we can make progress and move towards INT64 timestamps by default, but in the short term I still feel this change is targeted and useful.

There is also no current alternative for nano timestamps that need to span the SQL time RAnge (years 0001-9999)

[CurtHagenlocher]

There is also no current alternative for nano timestamps that need to span the SQL time RAnge (years 0001-9999)

As someone not really in the Spark world, doesn't the persistence of this type imply there's a need that Parquet isn't filling without it? Is it worth reconsidering its deprecation? I agree with @wgtmac that it's a bit weird to say "don't use this type! but if you do, then change your writers to emit this new sort order."

[etseidl]

For some (sometimes contentious) historical context, see

• INT96 should be marked as deprecated #301
• PARQUET-861: Document INT96 timestamps #49
• https://github.com/apache/parquet-format/pull/55/changes#r134738843
• PARQUET-1065: Deprecate type-defined sort ordering for INT96 type #77
• GH-502: Clarify Int96 status and add recommended ordering #504
• https://lists.apache.org/thread/6fm50b3pmh6mz659jb5wx5vzmvwccz1n

Together, they read like the stages of grief. I think this PR is "acceptance".

[divjotarora]

parquet-java reference implementation: apache/parquet-java#3610