[SPARK-57150][SDP] SCD1 Out-of-order Convergence Suite

[AnishMahto]

What changes were proposed in this pull request?

A key feature of SDP's AutoCDC implementation is that it supports reconciling out-of-order (by sequence) events. This support also adds significant complexity to the reconciliation logic as it requires cross-microbatch stateful tracking in the auxiliary table, and is prone to breaking as the implementation evolves over time.

Introduce an A/B style test suite to execute the implementation on both a sequence-sorted single-microbatch event stream and the same events on a shuffled multi-microbatch event stream. If out-of-order processing is correct, then the SCD1 implementation should produce the same target tables for both runs.

Data is randomly generated, but with a constant seed for reproducibility.

Why are the changes needed?

Test correctness of the AutoCDC SCD1 out-of-order reconciliation algorithm. Prevents signal for existing implementation's correctness and helps prevent regressions in future iterations.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Test only change.

Was this patch authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.7

[AnishMahto]

@jose-torres Short test-only PR for A/B testing the out-of-order correctness of the AutoCDC SCD1 implementaiton

[jose-torres]

Not sure this is quite right as a test structure. I'd expect to see a single test with a single seed that's randomized by default or uses a passed in command line argument. Right now there's no way to actually perform a reproduction without editing and rebuilding the test code.