Improve handling of quota being met when multiple instance types are specified in google batch executor #6686
Description
Enhancement
As it currently stands, when I specify multiple instance types, and a quota is hit for the first one, google batch will keep retrying it according to the error code set. For example:
machineType = 'n4*lssd,c4d*lssd,c4-*lssd'
When the quota for n4 instances is met, the rest of the instance types will not be attempted. It would be nice for all instances to be tried.
Use case
This would greatly improve resource handling in the google batch executor.
Suggested implementation
Implementation wise, looping through the possible instance choices automatically before throwing a pipeline ending failure/running the error handling logic would be ideal.