Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Bump minimum supported Spark runtime version to 3.2.0 and update unit tests #1480

Closed
parthosa opened this issue Dec 31, 2024 · 0 comments · Fixed by #1482
Closed

[BUG] Bump minimum supported Spark runtime version to 3.2.0 and update unit tests #1480

parthosa opened this issue Dec 31, 2024 · 0 comments · Fixed by #1482
Assignees
@parthosa
Labels
bug Something isn't working core_tools Scope the core module (scala)

Comments

@parthosa
Copy link
Collaborator

parthosa commented Dec 31, 2024 •
edited
Loading

Describe the bug
Currently, many of our AutoTuner unit tests recommend com.nvidia.spark.rapids.spark311.RapidsShuffleManager as the shuffle manager version. However, according to our latest documentation, Spark 3.1.1. is no longer supported.

The minimum supported versions are now:

  • com.nvidia.spark.rapids.spark320.RapidsShuffleManager for Spark distributions.
  • com.nvidia.spark.rapids.spark330db.RapidsShuffleManager for Databricks 11.3.

Expected behavior

  • We should remove the maven build profiles 3.1.1, 3.1.3 and 3.1.4 as these are no longer supported by plugin (Drop spark31x shims [databricks] spark-rapids#11159)
  • We should update AutoTuner unit tests to align with the minimum supported Spark and RapidsShuffleManager versions.
@parthosa parthosa added bug Something isn't working core_tools Scope the core module (scala) labels Dec 31, 2024
@parthosa parthosa self-assigned this Dec 31, 2024
@parthosa parthosa changed the title [BUG] Update AutoTuner unit tests to reflect minimum supported RapidsShuffleManager versions [BUG] Update AutoTuner unit tests to reflect minimum supported Spark and RapidsShuffleManager versions Dec 31, 2024
This was referenced Dec 31, 2024
Merged
Closed
@parthosa parthosa changed the title [BUG] Update AutoTuner unit tests to reflect minimum supported Spark and RapidsShuffleManager versions [BUG] Bump minimum supported Spark runtime version to 3.2.0 and update unit tests Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@parthosa parthosa

Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Milestone
No milestone
1 participant
@parthosa