Describe the feature
Add Diff incremental planning/reading: compare table state at two snapshots and emit the net changes (insert / update-before / update-after / delete) when no pre-materialized changelog is available.
Motivation
Many tables run with changelog-producer=none. Diff is the practical way to still produce an audit/changelog stream for batch incremental jobs.
Proposed scope (phase 1)
- Diff planning as per-(partition, bucket) before/after pairs
- Fail loud (no silent wrong answers) for:
- non-deduplicate merge engines
- bucket rescale between snapshots
- deletion vectors / nested/decimal types we can't safely compare yet
- Audit-style row kinds for the diff result
Depends on the base incremental scan + AuditLog read path.
Out of scope
- Full DV-aware diff
- Non-deduplicate merge engines
Describe the feature
Add Diff incremental planning/reading: compare table state at two snapshots and emit the net changes (insert / update-before / update-after / delete) when no pre-materialized changelog is available.
Motivation
Many tables run with
changelog-producer=none. Diff is the practical way to still produce an audit/changelog stream for batch incremental jobs.Proposed scope (phase 1)
Depends on the base incremental scan + AuditLog read path.
Out of scope