Optimize deferrable execution for AzureDataFactoryRunPipelineOperator

[Mohammed-Karim226]

Description

Currently, the AzureDataFactoryRunPipelineOperator defers tasks even if they have already been completed, leading to unnecessary deferring and potential inefficiencies. This can result in increased execution times and resource usage. The proposed optimization involves checking if a submitted job has already been completed before deferring it, thereby skipping deferral for already finished tasks.

Use case/motivation

This change will improve the efficiency and performance of the AzureDataFactoryRunPipelineOperator by reducing unnecessary deferring of tasks. It will help optimize resource usage and execution times, especially in environments with a high volume of tasks.

Related issues

N/A

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.