[Enhancement] [Build] Run Delta unit tests during PR validation

[felipepessoto]

Description

I think this can help avoiding regressions and bugs. During the build we could clone the Delta repo, https://github.com/delta-io/delta/, for each supported Delta version, for example, branch 3.2. apply a git patch to build.sbt to reference Gluten jars and change the DeltaSQLCommandTest to set the Gluten plugin in the sparkConf.

[zhztheplayer]

Thanks. This is a valid direction since we'd like to see the copied Delta unit tests can be finally removed from Gluten codebase.

Toward the proposed path, one challenge is there might be some unit tests that check the executed query plan, where Gluten might fail there. I don't have a good solution in mind but am also not so sure about the impact, how many tests are like that, etc.

[felipepessoto]

I think Delta don't have many unit tests checking the query plan. I need to think how we can get a manageable report of errors, but looking some random failures they look real issues, like

https://github.com/apache/gluten/actions/runs/27376177957/job/80905117213?pr=12278

incorrect results:

[info] - schema evolution - extra nested column in source - update, isPartitioned=true *** FAILED ***
[info]   == Results ==
[info]   !== Correct Answer - 2 ==                      == Spark Answer - 2 ==
[info]    struct<key:int,x:struct<a:int,c:int,b:int>>   struct<key:int,x:struct<a:int,c:int,b:int>>
[info]   ![1,[10,1000,100]]                             [1,[10,100,100]]
[info]    [2,[2,2000,null]]                             [2,[2,2000,null]] (QueryTest.scala:273)

[info] - Time travel with schema changes
[info] - data skipping still works with time travel *** FAILED ***
[info]   426 was not less than 426 (DeltaHistoryManagerSuite.scala:476)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)

failures:

[info] - create table *** FAILED ***
[info]   org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/__w/gluten/gluten/delta/spark/spark-warehouse/org.apache.spark.sql.delta.DeltaArbitraryColumnNameSuite/source. SQLSTATE: 58030
[info]   at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:814)
[info]   at org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$.executeTask(GlutenDeltaFileFormatWriter.scala:505)
....
[info]   Cause: java.util.concurrent.ExecutionException: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxUserError
[info] Error Source: USER
[info] Error Code: INVALID_ARGUMENT
[info] Reason: Field not found: col-3c1789ba-5bac-4efe-b631-cbd8b227dfbd. Available fields are: c with special chars ,;{}()
[info] 	=, d with special chars ,;{}()
[info] 	=.
[info] Retriable: False
[info] Context: Top-level Expression: (node_value-stream:0_1).col-3c1789ba-5bac-4efe-b631-cbd8b227dfbd
[info] Function: getChildIdx
[info] File: /work/ep/build-velox/build/velox_ep/velox/type/Type.cpp
[info] Line: 598
[info] Stack trace:
[info] # 0  
[info] # 1  
[info] # 2  

Failures instead of fallback:

[info] - Variant type *** FAILED ***
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1489.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1489.0 (TID 1267) (1c419bb89a89 executor driver): java.lang.UnsupportedOperationException: Unsupported data type: variant
[info] 	at org.apache.spark.sql.utils.SparkArrowUtil$.toArrowType(SparkArrowUtil.scala:60)
[info] 	at org.apache.spark.sql.utils.SparkArrowUtil$.toArrowField(SparkArrowUtil.scala:137)
[info] 	at org.apache.spark.sql.utils.SparkArrowUtil$.$anonfun$toArrowSchema$1(SparkArrowUtil.scala:167)

[zhztheplayer]

@felipepessoto

If the failures reflect Gluten's bugs, we may need a way to ignore them and maintain an ignored case list in Gluten's code base?