Fix "Deadlock in Scheduler While Handling High Concurrency DAGs"

[shreyashdwivedi36]

Apache Airflow version

2.10.4

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The scheduler stops processing tasks due to a deadlock in the resource allocation mechanism. Logs show repeated warnings about locking mechanisms and no progress in DAG processing.

What you think should happen instead?

The scheduler should efficiently handle high concurrency workloads without encountering deadlocks. Tasks should progress as expected, and the system should avoid resource contention issues.

How to reproduce

1. Deploy Apache Airflow 2.7.0 with a high-concurrency configuration.
2. Submit a complex DAG with multiple tasks requiring concurrent execution.
3. Observe the scheduler logs for deadlock warnings and stalled task processing.

Operating System

Ubuntu 20.04

Versions of Apache Airflow Providers

Run the following command in your environment where Apache Airflow is installed:

pip freeze | grep apache-airflow-providers

Deployment

Docker-Compose

Deployment details

1. Airflow Deployment Type: Deployed using Docker Compose with the official Airflow Docker image.
2. Environment Configuration: Includes PostgreSQL as the metadata database and Redis as the Celery backend.
3. Customization: Custom configuration for logging and email alerts added to airflow.cfg.
4. Version of Deployment Tools:

• Docker: 20.10.14
• Docker Compose: 2.12.2

Anything else?

Frequency of the Problem: The issue occurs consistently under high concurrency scenarios.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

You report is confusing and ambiguous. You need to add the logs and dag example to make it more concrete and reproducible.