Apache Iceberg version
No response
Query engine
Kafka Connect
Please describe the bug 🐞
Summary
When using the Iceberg Kafka Connect sink with a primary key (equality field IDs) (a, b, ...), a crash occurring between two consecutive updates of the same primary key can result in both versions of the row living in the target table.
The root cause is that both the old and the new versions of the row end up being committed within the same snapshot. Because an equality delete only removes data rows from previous (lower sequence number) snapshots and has no effect on data rows added in the same snapshot, the equality delete fails to remove the stale row. As a result, the target table ends up with duplicate rows for the same primary key.
Scenario / Steps to reproduce
- A record with primary key
(a, b, ...) is updated. The sink writes the new data row plus an equality delete for the old value.
- A crash happens before the offsets/commit are finalized, so the work is retried.
- The same primary key
(a, b, ...) is updated once again.
- Both updates (the data rows and their equality deletes) are flushed and committed together in a single snapshot.
- Query the target table.
Expected behavior
Only the latest version of the row for primary key (a, b, ...) should exist in the target table.
Actual behavior
Both versions of the row for primary key (a, b, ...) are present in the target table. Since both are committed in one snapshot, the equality delete has no effect on the row added in the same snapshot (equality deletes only apply to data files with a lower sequence number).
Notes
This appears to be a consequence of equality-delete semantics: an equality delete with sequence number S only deletes data rows with sequence number < S; rows added in the same snapshot (same sequence number) are not affected. When a crash forces multiple PK updates to be batched into one snapshot, the intermediate stale version is not removed.
Willingness to contribute
Apache Iceberg version
No response
Query engine
Kafka Connect
Please describe the bug 🐞
Summary
When using the Iceberg Kafka Connect sink with a primary key (equality field IDs)
(a, b, ...), a crash occurring between two consecutive updates of the same primary key can result in both versions of the row living in the target table.The root cause is that both the old and the new versions of the row end up being committed within the same snapshot. Because an equality delete only removes data rows from previous (lower sequence number) snapshots and has no effect on data rows added in the same snapshot, the equality delete fails to remove the stale row. As a result, the target table ends up with duplicate rows for the same primary key.
Scenario / Steps to reproduce
(a, b, ...)is updated. The sink writes the new data row plus an equality delete for the old value.(a, b, ...)is updated once again.Expected behavior
Only the latest version of the row for primary key
(a, b, ...)should exist in the target table.Actual behavior
Both versions of the row for primary key
(a, b, ...)are present in the target table. Since both are committed in one snapshot, the equality delete has no effect on the row added in the same snapshot (equality deletes only apply to data files with a lower sequence number).Notes
This appears to be a consequence of equality-delete semantics: an equality delete with sequence number
Sonly deletes data rows with sequence number< S; rows added in the same snapshot (same sequence number) are not affected. When a crash forces multiple PK updates to be batched into one snapshot, the intermediate stale version is not removed.Willingness to contribute