Core: Add content stats evaluator - #17413
Conversation
github-actions
Bot
added
API
core
Specification
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.iceberg.expressions; |
There was a problem hiding this comment.
I think this forced promotion of package private v4 class into public and result in the revapi breakage. I am wondering if we shall consider the alternative below ?
- increase the visibility of InclusiveEvalVisitor into public
- move
InclusiveStatsEvaluatorto org.apache.iceberg package with package private class.
This avoid the unnecessary change to ContentStats, FieldStats and TrackedFile and revapi. We can eventually move InclusiveStatsEvaluator to org.apache.iceberg.expressions package once it's v4 ready.
This also require expose VariantExpressionUtil.castTo() as a protected method in InclusiveEvalVisitor.
|
|
||
| Field-level structs in `content_stats` are based on the corresponding table field's type, requirement, and ID (`field-id`). | ||
|
|
||
| Stats are tracked for primitive and `variant` fields. Stats are not tracked for `struct`, `list`, and `map` fields, or for any field contained in a `list` or `map` because it may be repeated. |
There was a problem hiding this comment.
maybe consider move the spec to a separate PR?
There was a problem hiding this comment.
yep that's the plan. I'm currently also looking at an approach that wouldn't require any spec changes
| | _optional_ | 3 | `tight_bounds` | `boolean` | all except `geometry`, `geography`, `variant` | When true, `lower_bound` and `upper_bound` must be equal to the min and max values | | ||
| | _optional_ | 4 | `value_count` | `long` | all | Number of values in the column (including null and NaN values) | | ||
| | _optional_ | 5 | `null_value_count` | `long` | optional fields | Number of null values in the column | | ||
| | _optional_ | 5 | `null_value_count` | `long` | fields that may be null | Number of null values in the column | |
There was a problem hiding this comment.
looks like we now omit the fields which is definitely required, null count is guaranteed to be 0. Curious, is this intentional depart from v1-v3 null_count?
| } | ||
|
|
||
| return false; | ||
| } |
There was a problem hiding this comment.
I think this method is very similar to InclusiveStatsEvaluator::allAncestorFieldsAreRequired, since both are purely schema based and have nothing to do with stats, maybe consider move to the schema.isNullable(int fieldId) instead as it already have the fieldId of all schema cached lazily?
In reader, it can use referencedIds.stream().filter(id -> !schema.isNullable(id)) and in writer it can be fieldStatsStruct(tableSchema.isNullable(id), ...).
There was a problem hiding this comment.
yep, I've unified the functionality already into TypeUtil.isNullable()
The single_value_nulls fixtures in testNotEqWithSingleValue and testNotInWithSingleValue reported 2 nulls for field 3, the "required" column. A required column cannot contain nulls, so no writer produces those metrics. Move both to the optional "some_empty" column, which keeps the case the tests cover: a file whose bounds hold a single value still has to be read when the column also contains nulls, because those nulls match != and NOT IN. The NOT IN fixture also reported a NaN count for a string column, which only float and double columns track.
The range_of_values and single_value fixtures reported a NaN count for field 3, a string column. Only float and double columns track NaN counts, which is why the single_value fixture in testNotEqWithSingleValue already passed null for them. Generated-by: Cursor
`FILE_6` in `TestInclusiveMetricsEvaluator` carried value and null counts for the
`required_address` and `optional_address` structs. Parquet produces metrics for leaf
columns only in `ParquetMetrics.MetricsVisitor.struct` and returns just the concatenation of
its children and never emits a `FieldMetrics` for the struct — so no Parquet file yields
those entries. They made `notNull("optional_address")` appear prunable when in practice it is not
…luatorWithExtract/TestInclusiveMetricsEvaluatorWithTransforms extendable
a4e91ed to
d505d89
Compare
Content stats omit null_value_count for a field that cannot be null, but InclusiveStatsEvaluator read the missing count as unknown and stopped pruning files for notEqual, notIn, and notStartsWith. The v3 metrics maps carry an explicit zero from Parquet, so the same filters pruned there. Detect the fields that cannot be null the same way expression binding does, by requiring the field and all of its ancestors to be required, and report no nulls for them.
The content stats schema omitted null_value_count whenever a field was required, but a required field nested in an optional struct is null on every row where its parent is null, and Parquet reports that count from the leaf column's definition levels. Include the count for any field that may be null, either because it or one of its ancestors is optional.
d505d89 to
1f62852
Compare
2 participants
No description provided.