Add qualx support for platform runtime variants (DB AWS)

[leewyang]

This PR follows #1414 to support variants of platform runtimes (e.g. photon) in the qualx models.

Model variants are delimited by the underscore character '', i.e. .
This PR also updates the models from the latest code and datasets, including the new databricks-aws_photon model.

Changes

1. adds sparkRuntime as a new expected_raw_feature.
2. uses the sparkRuntime column to detect datasets with mixed runtimes (i.e. more than one).
3. modifies the prediction loop to operate on input rows grouped by runtime variant.

Test

Following CMDs have been tested:

spark_rapids prediction

Internal Usage:

python qualx_main.py preprocess
python qualx_main.py train
python qualx_main.py evaluate

[parthosa]

Thanks, @leewyang! This PR introduces Photon models for Databricks on AWS. Could we update the title to reflect “DB AWS”?

In future, when we add support for DB Azure models, this will help provide clarity.

[parthosa]

Thanks @leewyang for adding the Photon model.

[tgravescs]

so DB AWS here is in the title because we use a model training on eventlogs from databricks aws? What does it do on azure or gcp? Are we expecting differences there? I would expect the databricks code to the same but you have different machine types and possibly I/O characteristics so I'm wondering if we have seen differences from those.

[parthosa]

so DB AWS here is in the title because we use a model training on eventlogs from databricks aws?

Yes, it adds a model trained on DB AWS Photon event logs.

What does it do on azure or gcp? Are we expecting differences there?

• Model for Azure has been added in PR: Add model for databricks-azure_photon and update combined model #1427
• The code looks for a model named: <platform><runtime>
• If the model file does not exist, it loads the default <platform> only model file.