Skip to content

fix(transforms): day transform result type is date, not int#1176

Closed
twuebi wants to merge 1 commit into
apache:mainfrom
twuebi:tp/fix-partition-dtype-mapping
Closed

fix(transforms): day transform result type is date, not int#1176
twuebi wants to merge 1 commit into
apache:mainfrom
twuebi:tp/fix-partition-dtype-mapping

Conversation

@twuebi

@twuebi twuebi commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

DayTransform.ResultType returned Int32, diverging from Java, Rust, and PyIceberg, which all return date. The physical value stays int32 days, the date type adds the date Avro logicalType to the partition field, so day partitions round-trip with the same logical type Spark writes.

Without it, compacting a days partitioned table whose mainfest was written by spark failed on commit:

avro: field data_file.partition.ts_day: cannot use time.Time with Avro type intavro: field data_file.partition.ts_day: cannot use time.Time with Avro type int (*fmt.wrapError)

@twuebi twuebi requested a review from zeroshade as a code owner June 9, 2026 14:50
@laskoviymishka

Copy link
Copy Markdown
Contributor
image

@laskoviymishka

Copy link
Copy Markdown
Contributor

@laskoviymishka

Copy link
Copy Markdown
Contributor

One more point to connect dotes - apache/iceberg#16446

@twuebi

twuebi commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

oh, sorry for not checking issues / open PRs first, this blew up compaction running on a table with partition by day written by spark

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants