Bump minimum Spark version to 3.2.0 and improve AutoTuner unit tests for multiple Spark versions

[parthosa]

Fixes #1480

This PR removes the Maven build profiles for Spark versions 3.1.1, 3.1.3, and 3.1.4, excludes Spark 3.1.1 from the GitHub workflow, and updates the AutoTuner unit tests to dynamically test against multiple Spark versions.

Detailed Changes

1. Build and Workflow Cleanup:

   • Removed outdated Maven build profiles for Spark 3.1.1, 3.1.3, and 3.1.4.
   • Excluded Spark 3.1.1 from the GitHub workflow.

2. AutoTuner Unit Tests:

   • Updated to dynamically determine the Spark runtime version using ToolUtils.sparkRuntimeVersion.
   • Enables testing across different Spark versions in GitHub workflows without hardcoding specific versions.
   • Modified tests to recommend minPartitionSize for Spark 3.2.0+ instead of the deprecated minPartitionNum.
     • Reference: Spark SQL Performance Tuning Documentation.

[amahussein]

Thanks @parthosa !
Just wondering why bumping it to an old version while we can leap to a more recent job.

[amahussein]

Should we just bump it to the most recent? something like 3.5.x?

[parthosa]

That makes sense. I have updated the default Spark version in the AutoTuner tests to 3.5.1.

This also brings up an idea (if needed in future): we could test for both the lower and upper bounds to ensure compatibility across the entire supported range.

[amahussein]

Thanks @parthosa. That's a good idea.
We can actually read the Spark version from runTime. Doing it dynamically based on the pom will achieve what you just suggested. Then the test will be done for each spark-version in the github workflow.
We can get the spark version using 2 methods:

• From pom file. We can do that by pulling the build.spark.version; or doing the easy way
• by getting ToolUtils.sparkRuntimeVersion

[parthosa]

Thanks @amahussein for the suggestion. I think that would be the right thing to do. In that case, we should drop the mvn profiles (<3.2.0). There is a similar issue NVIDIA/spark-rapids#11160 on the plugin side to drop tests for versions <3.2.0.

[parthosa]

Summary from offline discussion:

• We should remove the Maven build profiles for Spark versions 3.1.1, 3.1.3, and 3.1.4
• Update the AutoTuner unit tests to dynamically test against multiple Spark versions.

[amahussein]

Is this working? I thought that we cannot use spark-4.0.0 yet as it is based on scala 2.13.

• the maven repository has spark-4.0.0-previews only https://mvnrepository.com/artifact/org.apache.spark/spark-core
• the archive links also do not have scala2.12 binaries in the spark-4.0.0. https://archive.apache.org/dist/spark/spark-4.0.0-preview2/

[parthosa]

We added support Spark 4.0.0 in #537. This change adds it to github workflow. All tests are passing on Spark 4.0.0.

[amahussein]

Yes, you are right. We added it long time ago.
Then we removed it from workflow when the scala2.12 was dropped.
I believe this success is bogu because it fetches an outdated snapshot of Spark4.0.0 that was compatibe with scala2.12. It is dated september 2023
https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-core_2.12/

But if you look closely to scala2.13, you will see that up-to-date artifacts are January 2025
https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-sql_2.13/

We should remove that 400 from the github-workflow because it is not really testing Spark-400

[parthosa]

Ahh. I see. Thank you for this catch. I have removed it from the workflow.

[amahussein]

Thanks @parthosa !
LGTME