Adding max_retries for ECSOperator

[markhopson]

Would it be valuable to add a max_retries param to the ECSOperator?

In my use-case, I have a DAG that uses the ECSOperator to start a Task, but sometimes my ECS cluster is at capacity and the ECSOperator fails. Instead, if the ECSOperator retried a couple of times (say 3 times over the period of 5 mins), it would give my ECS cluster time to scale-out and accomodate the Task being created by the ECSOperator.

What does everyone think?

BTW I'm a first time poster here, and am looking forward to contribute to this project.

[potiuk]

Sure. Go for it. We have similar retry mechanisms implemented in 'google" provider.

This is especially useful if you have a way to distinguish the "capacity" (transient) error from any other "permanent" one):

See here for example (this is "quota_retry" but there are few others):

airflow/airflow/providers/google/common/hooks/base_google.py

Line 359 in 2f79fb9

def quota_retry(*args, **kwargs) -> Callable:

We used tenacity to provide a capability of exponential back-off for such retries and we recommend the same approach - this way you are even better in handling "big" spikes.

We implemented it as decorators, so it could be applied to a wide range of Google Operators (and some Google APIs have built-in Retry capability in which case we used the built-in ones. Maybe you could also work out similar pattern for many Amazon operators?

[markhopson]

Thanks for the quick (and helpful) response. I'll take a look at the existing pattern in the Google Operators.

If I have any questions, where should I post them?

Thanks again!

[potiuk]

Might be here.

[markhopson]

Finally got a chance to look at this over the weekend, and it seems that boto3 does not have a consistent way of returning errors across all AWS services.

However, boto3 does seem to be consistent in returning errors across ECS APIs. It seems like ECS APIs like to return a response with a failures param that stores an Array of "error" Objects I can parse.

If all this is correct, it feels like my decorator should be ECS specific - so I would not define in base_aws, and instead, put it in ecs.py?

What do you think?