Supervisor: Limit the number of parallel evaluations

[jenny011]

Currently, if we use fixed dataset in pipeline_config and disable evaluation_matrix, then pipeline_executor starts the evaluations all at once. Depending on the size of the eval dataset, we can run out of gpu memory.
If necessary, we can add a knob to run the evaluations in smaller batches.

[robinholzi]

@MaxiBoether Do you still know of any issues in that regard? I think we can close this.