[FEA] Add "preserve" and "exclude" sections to target cluster

[parthosa]

Problem Statement

Currently, AutoTuner's target cluster configuration only supports an "enforced" section for overriding property values. However, in some cases users might require the following:

1. Preserving Source Values: Users want to maintain certain Spark property values exactly as they were in the original application
2. Excluding Properties: Users want to exclude certain Spark properties from recommendations due to environment constraints or incompatibilities

Solution

Add two new sections to the target cluster configuration, similar to the existing "enforced" section:

1. "preserve" Section
   • Type: List of property names
   • Behavior: Uses values from the source application and treats them as enforced
2. "exclude" Section
   • Type: List of property names
   • Behavior: Excludes properties from tuning recommendations

YAML Configuration Example

sparkProperties:
  enforced:
    spark.executor.cores: "8"
    spark.executor.memory: "16g"
  preserve:
    - spark.sql.shuffle.partitions
    - spark.sql.files.maxPartitionBytes
    - spark.sql.adaptive.coalescePartitions.initialPartitionNum
  exclude:
    - spark.rapids.shuffle.multiThreaded.reader.threads
    - spark.sql.adaptive.autoBroadcastJoinThreshold
  tuningDefinitions:
    # ... existing tuning definitions