Airflow jobs are sporadically re-queuing days after dags are completed

[cjac]

Apache Airflow version

2.2.3

What happened

The queuing process in Airflow2 continues to access
already-completed job, effectively corrupting start times/end
times. This ruins the ability for job dependencies to chain together.

After a long period of no change to completed tasks, while under heavy load, airflow re-schedules all historical jobs just to have them do no operation and complete. When the job is re-queued, the start time is re-set, and nothing happens, and the end time is immediately re-set as well.

The updating of these dates to NOW() is problematic, since there are listeners monitoring the values of these fields. When the completion time is updated, other tasks are fired off.

Why did the jobs get re-queued?

We are able to reproduce the problem and provide a video call where questions may be asked.

What you expected to happen

Once tasks are complete, they should never be re-queued and re-executed.

If tasks must be re-queued and re-executed, the start and end times of the already completed tasks must not be updated.

How to reproduce

Put environment under heavy load for a short period
monitor the start/end times of tasks
note that the start/end times are updated all concurrently

Operating System

composer-2.0.3-airflow-2.2.3

Versions of Apache Airflow Providers

composer-2.0.3-airflow-2.2.3

Deployment

Composer

Deployment details

Available on request from customer. Please be discrete with these sensitive information.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

This has no reproducible case so it is at most a discussion. It's impossible to say what the problem is without looking at your DAGs, but what happens most likely is either you have some historical data cleanup that removes DAG execution history. It could also be DAG definition using dynamic start_date (we are just about to complete changing all our examples to stop using "days_ago" in start_date with similar observed (and confusing behaviour). Please review your DAGs according to the https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html and examples there. And look whether there is no historical database pruning implemented by Composer or in your deployment that could trigger itl

There is not much anyone can help with without you providing detailed information on your DAGs.

[cjac]

Thanks potiuk,

I'll share this with customer and report back tomorrow. We may be able to strip PII from customer's DAG and share portions. Would you be willing and able to work with us if we are able to share relevant information?

[potiuk]

No. Obviously I am not.

This is not how it works here. You get what you pay for when you get free software. We help here as voluntaries. Any help here is done on "as is" basis and might or might not come from me or from anyone else who wills to help.

If you want someone to commit to help - hire that person or company and pay for support (and with expert knowledge like that the rates are high usually). If you want I can recommend you quite a few companies that provide paid support (and you can easily find it).

[cjac]

I see. Thank you for being forthright. I want to help your team to help us. I will see if we can gather a simple reproduction, and I will review your test suite to see if I can create a unit test which reproduces the error.

[SteveScott]

I had a similar issue. Airflow scheduler and webserver both were down. I restarted them and all my workflows kicked off simultaneously. I want to prevent this.