DAGs disappear from web UI

[sarthakhaikerwalsaxena]

We are facing an issue where ALL DAGs sometimes disappear from web UI and then become visible again automatically. We are running two webservers and schedulers for HA.
The issue is reproducible easily if we refresh 10-15 times and ALL the dags disappear at the time of the issue. Can someone help with the possible cause of the issue? Thanks.

[potiuk]

1. likely it is because you have non-atomic refreshes of your filesystem's DAGS folder. If there is a case when your synchronization will produce "partial" update wherere some DAGs will be missing during sync, you will have similar problem. Fix your sync solution to be atomic (similarly to what Airlow Helm Chart git-sync provides). This is the best solution for now.

2. I think in the future, this might be somewhat mitigated by this coming change Reduce DB load incurred by Stale DAG deactivation #21399 (but it wil depend on your configuration so it might, or might not help.

[grv-trip]

@potiuk we are running into a similar issue with dags disappearing (and dags run affected by this), with git-sync v3.5 and airflow 2.2.5 (put up by official airflow helm chart). In fact, we are using git-sync, where we are syncing the Github dags repository into EFS. EFS is mounted inside the Airflow scheduler. Also, the sync is happening every 300 seconds.

When you refer to atomic refreshes, can you please elaborate on this? I am trying to ascertain if we have missed something trivial here in our setup

@sarthakhaikerwalsaxena Did you ever solve this problem?

[potiuk]

Then EFS is likely your problem. It caused others problems, especially when they have not bought extra IOPS.

It makes very little sense to have EFS and Git-Sync together because when git-synced replaces DAGs atomically with symbolic link replace, it basically means that EFS must re-distribute all the DAG files to all the workers and schedulers over the network and while doing it (possibly with limited number of IOPS) only some files are visible to scheduler and it might delete them,

There are multiple approaches:

1. Buy extra IOPS for EFS - that seemed to help some users when they used EFS

2. never combine GitSync and EFS. It makes no sense IMHO.

Run Git-Sync container on each of your components and don't use DAG persistency. You do not need dag persistency when you run GitSync for all containers. This is known to bring back rock-solid performance.

[mewa]

@potiuk I think this might be an issue with how the dags are updated.

We also ran into this problem and the interval at which dags start to re-appear seems to be linked to min_serialized_dag_update_interval (30 by default).

Also, they start disappearing roughly around the interval specified by min_serialized_dag_fetch_interval (10 by default).

Looks like more people are having this problem as well.

[potiuk]

I suggest @mewa you upgrade to lates Airflow version and create an issue describing it with all details including some attempt to get a reproducible scenario. It's impossible to act on it at this stage without enough details.