Scheduler crash loop in GCP Composer

[alucryd]

Hi all,

We are in a bit of a bind, we have 2 identical Composer environments on GCP (composer-2.9.4-airflow-2.9.3), one of them suddenly stopped working and the scheduler is now in a crash loop. No amount of DAG run or Task instance deletion helped. Both environments run the same DAGS with the same scheduling. Here is the stack trace:

2024-09-27 09:20:33.228 CEST airflow-scheduler Traceback (most recent call last): File "/opt/python3.11/bin/airflow", line 8, in <module> sys.exit(main()) ^^^^^^
2024-09-27 09:20:33.229 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/__main__.py", line 58, in main
2024-09-27 09:20:33.230 CEST airflow-scheduler args.func(args)
2024-09-27 09:20:33.230 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command
2024-09-27 09:20:33.231 CEST airflow-scheduler return func(*args, **kwargs)
2024-09-27 09:20:33.231 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.231 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/cli.py", line 114, in wrapper
2024-09-27 09:20:33.232 CEST airflow-scheduler return f(*args, **kwargs)
2024-09-27 09:20:33.233 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.233 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
2024-09-27 09:20:33.233 CEST airflow-scheduler return func(*args, **kwargs)
2024-09-27 09:20:33.235 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.235 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 58, in scheduler
2024-09-27 09:20:33.237 CEST airflow-scheduler run_command_with_daemon_option(
2024-09-27 09:20:33.237 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/commands/daemon_utils.py", line 85, in run_command_with_daemon_option
2024-09-27 09:20:33.239 CEST airflow-scheduler callback()
2024-09-27 09:20:33.239 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 61, in <lambda>
2024-09-27 09:20:33.240 CEST airflow-scheduler callback=lambda: _run_scheduler_job(args),
2024-09-27 09:20:33.246 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.246 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 49, in _run_scheduler_job
2024-09-27 09:20:33.247 CEST airflow-scheduler run_job(job=job_runner.job, execute_callable=job_runner._execute)
2024-09-27 09:20:33.247 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/session.py", line 79, in wrapper
2024-09-27 09:20:33.249 CEST airflow-scheduler return func(*args, session=session, **kwargs)
2024-09-27 09:20:33.250 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.250 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/job.py", line 402, in run_job
2024-09-27 09:20:33.252 CEST airflow-scheduler return execute_job(job, execute_callable=execute_callable)
2024-09-27 09:20:33.253 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.253 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/job.py", line 431, in execute_job
2024-09-27 09:20:33.254 CEST airflow-scheduler ret = execute_callable()
2024-09-27 09:20:33.255 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.255 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 846, in _execute
2024-09-27 09:20:33.325 CEST airflow-scheduler self._run_scheduler_loop()
2024-09-27 09:20:33.326 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 978, in _run_scheduler_loop
2024-09-27 09:20:33.328 CEST airflow-scheduler num_queued_tis = self._do_scheduling(session)
2024-09-27 09:20:33.330 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.330 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1052, in _do_scheduling
2024-09-27 09:20:33.335 CEST airflow-scheduler self._create_dagruns_for_dags(guard, session)
2024-09-27 09:20:33.335 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/retries.py", line 89, in wrapped_function
2024-09-27 09:20:33.336 CEST airflow-scheduler for attempt in run_with_db_retries(max_retries=retries, logger=logger, **retry_kwargs):
2024-09-27 09:20:33.336 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 443, in __iter__
2024-09-27 09:20:33.338 CEST airflow-scheduler do = self.iter(retry_state=retry_state)
2024-09-27 09:20:33.342 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.342 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 376, in iter
2024-09-27 09:20:33.343 CEST airflow-scheduler result = action(retry_state)
2024-09-27 09:20:33.345 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.345 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
2024-09-27 09:20:33.346 CEST airflow-scheduler self._add_action_func(lambda rs: rs.outcome.result())
2024-09-27 09:20:33.347 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.347 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/concurrent/futures/_base.py", line 449, in result
2024-09-27 09:20:33.348 CEST airflow-scheduler return self.__get_result()
2024-09-27 09:20:33.348 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.348 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
2024-09-27 09:20:33.349 CEST airflow-scheduler raise self._exception
2024-09-27 09:20:33.349 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/retries.py", line 98, in wrapped_function
2024-09-27 09:20:33.350 CEST airflow-scheduler return func(*args, **kwargs)
2024-09-27 09:20:33.355 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.355 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1126, in _create_dagruns_for_dags
2024-09-27 09:20:33.356 CEST airflow-scheduler self._create_dag_runs_dataset_triggered(
2024-09-27 09:20:33.356 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1279, in _create_dag_runs_dataset_triggered
2024-09-27 09:20:33.424 CEST airflow-scheduler data_interval = dag.timetable.data_interval_for_events(exec_date, dataset_events)
2024-09-27 09:20:33.426 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.426 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/timetables/simple.py", line 194, in data_interval_for_events
2024-09-27 09:20:33.427 CEST airflow-scheduler start = min(start_dates)
2024-09-27 09:20:33.427 CEST airflow-scheduler ^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.427 CEST airflow-scheduler TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'

DAG start dates are mostly days_ago(1) with the occasional hardcoded date.

Any idea?

[MatteoEgmont]

Hi @alucryd , we are in the same situation as you are, running composer 2.9.6 with airflow 2.9.3
Did you manage to find a way to fix it?

[alucryd]

Hi @MatteoEgmont, I haven't but I think I found the culprit at least. Do you use dataset scheduling? We have a mix of cron, daily, weekly, hourly scheduling for our upstream dags, but all our downstream dags use dataset scheduling. As soon as we remove the dataset scheduling in favor of a cron scheduling, the scheduler comes back to life, and stays alive.

[MatteoEgmont]

Hiya! Yes indeed, we figured that out last night after a bit of trial and error to mitigate the troubles 😅 .
Removing the only one dag we had with dataset scheduling solved it. GCP changes in the managed part I imagine, it does not feel like an Airflow only issue?
Thanks a lot for getting back to us, much appreciated!

[alucryd]

Not sure yet whether this is an upstream issue or a composer issue, we are now in touch with Google so they can investigate.