[AURON #2429] Add mixed-format fallback coverage for Iceberg changelog native scan

[weimingdiit]

Which issue does this PR close?
Closes #2429

Rationale for this change
Auron native Iceberg changelog scan requires all planned changelog tasks to use a single supported file format.

Without validating the planned tasks, a mixed-format fallback test could pass because of an unrelated changelog fallback and fail to exercise the intended file-format guard.

What changes are included in this PR?
Adds fallback coverage for an Iceberg changelog range containing both Parquet and ORC data files.

The test:

• verifies that all planned changelog tasks are AddedRowsScanTask
• verifies that the task file formats are exactly Parquet and ORC
• compares the result with Spark execution
• verifies that NativeIcebergTableScan is not used

No execution logic is changed.

Are there any user-facing changes?
No user-facing changes. This is a test-only change.

How was this patch tested?
UT.

[weiqingy]

I went looking at the CI log for this commit to see which guard actually catches this case, and the delete seems to land on the metadata path rather than merge-on-read: the DELETE FROM commits as StreamingDelete with removedDataFiles=CounterResult{unit=COUNT, value=1}, totalDeleteFiles=CounterResult{unit=COUNT, value=0} and addedPositionalDeleteFiles=null, so no delete file gets written. The two-row insert above landed in two data files (The input RDD has 2 partitions, Committing append with 2 new data files), so id = 1 covers a whole file and Iceberg can drop it by metadata regardless of write.delete.mode.

If that reading is right, the fallback is coming from the task-type check at IcebergScanSupport.scala:266, the same one the existing "falls back when delete changes exist" test hits, and the delete-file check just below at :274 is still waiting for a test. Does that match what you were going for, or were you hoping this one would reach :274?

The test looks like it earns its place either way: it's the only one where added and deleted tasks share a single changelog range, which a guard like addedRowsTasks.isEmpty would get wrong. That just leaves the merge-on-read property as a loose end. Is it worth keeping when it isn't changing the outcome?

On the name, same thread of thought: "unsupported changelog operations" sent me looking for the operation() != INSERT check at :270, but AddedRowsScanTask.operation() in Iceberg 1.10.1 is a default returning INSERT that neither BaseAddedRowsScanTask nor its split subclass overrides, so after the collect at :263 I don't think it can fire. Would something like "falls back when a changelog range mixes inserts and deletes" capture the case better?

[weimingdiit]

Yes, that matches what is happening here. This case does not reach the delete-file guard. The fallback comes from the changelog task-type check because the changelog range contains both added-row tasks and non-added-row tasks.

I renamed the test to "iceberg changelog scan falls back when a changelog range mixes inserts and deletes", removed the misleading merge-on-read delete property, and added assertions that the planned changelog tasks contain both AddedRowsScanTask and non-AddedRowsScanTask. That makes the intended coverage explicit.

[weiqingy]

Thanks for confirming.

I don't see the new commit yet (branch is still on 713c2018), so this is off your description rather than the code.

Worth settling before the rename lands: #2435 makes this exact range native. Its toNativeChangelogDataFileTask accepts a DeletedDataFileScanTask with operation DELETE and empty existingDeletes(), which is what this whole-file delete produces, so all three tasks in the range (two AddedRowsScanTask, one DeletedDataFileScanTask, all PARQUET) convert and assert(!plan.contains("NativeIcebergTableScan")) flips. #2435 hit this on the sibling test and rewrote "falls back when delete changes exist" into "supports mixed insert and full-data-file delete changelog scan", but it branched off master so this test isn't in its tree.

So the two PRs would describe the same scenario with opposite expectations, and merge order decides which one breaks. Which way should it go: #2435 updates this test too, or it gets reshaped here to still fall back? The second may not be available. In Iceberg 1.10.1 the changelog scan only builds BaseAddedRowsScanTask and BaseDeletedDataFileScanTask, both of which #2435 accepts, and orderedChangelogSnapshots throws UnsupportedOperationException("Delete files are currently not supported in changelog scans") once a snapshot in range carries a delete manifest. If that is right, nothing reaches the task-type check afterwards. Does that match your reading?

[weimingdiit]

Yes, that matches my reading after checking #2435 and the Iceberg 1.10.1 changelog planning path.

For a planable changelog scan, Iceberg 1.10.1 creates AddedRowsScanTask for added data-file entries and DeletedDataFileScanTask for deleted data-file entries. Both are created with empty delete lists. If a snapshot contains delete manifests, orderedChangelogSnapshots throws before Auron receives any planned changelog tasks.

Therefore, after #2435, all tasks produced by this test are supported: the added tasks are INSERT tasks with empty deletes, the deleted task is a DELETE task with empty existingDeletes(), and all files are Parquet. The fallback expectation in #2430 would no longer be valid.

I will remove the mixed insert/delete fallback test from #2430 and keep the mixed-file-format fallback coverage. #2435 already covers the mixed insert and full-data-file delete range with the positive native expectation, so keeping that scenario there avoids merge-order-dependent tests.

I will also update the PR description to remove the unsupported-operation fallback claim. Thanks for catching the conflict.

[weiqingy]

Thanks for the quick turnaround. My thread is closed.

One thing I checked while looking at the split: the mixed-format test that stays doesn't have the same collision. #2435 keeps formats.size > 1, just over nativeChangelogTasks now, and its toNativeChangelogDataFileTask still accepts these AddedRowsScanTasks, so the range converts and then falls back on format either way. Nothing merge-order-dependent left as far as I can see.