Airflow 2.x schedule availability

[keith6014]

I run airflow 2.3.0 scheduler on a baremetal box. The database is on GCP (postgresql) -- assume this is already high availability. I would like to have multiple schedulers if my primary scheduler is down. I need to preserve state of my DAGs. I understand Airflow 2.x and above has high availability scheduler. Where can I read more about it and some gotchas.

[Taragolis]

@keith6014

Where can I read more about it and some gotchas.

• https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html?highlight=scheduler#running-more-than-one-scheduler
• https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#use-row-level-locking

I run airflow 2.3.0

My personal recommendation also switch to latest patch version of 2.3. You could see a lot of bug fixes in Release Notes.

[potiuk]

Absolutely. Actually we just released Airflow 2.4.0, so you should rather try 2.4.0 than 2.3.0. It makes little sense to use 2.3.0

[keith6014]

@potiuk and @Taragolis
So for my DAGs folder, should I keep them in a central location or make duplicate copies on each scheduler? How would that work?

[potiuk]

Pick your poison: https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html - this is for helm chart, but it is a good overview of what you can do. But you are the only person who can decide how to do it.

[keith6014]

I was thinking of using NFS to share my DAGs. For my 2 nodes I plan to expose dags_folder=/media/dags.
Should I run multiple webservers. I was thinking of running webserver on port :80 on both nodes. How would the webserver artitecture be?

[potiuk]

I think this is a question you have to answer yourself. As a deployment manager of your Airflow, you need to understand what you want to achieve, who and how will be using it, what properties of the system are important to you - and make decisions accordingly.

There are no "best" answers from Airflow perspective. Airflow van run multiple of everything, so you can run them as you want (see the docs) but whether you want to run any multiples, you need to decide based on what you want to achieve.

I don't think anyone can answer here anythin more than what you can already find in the docs: https://airflow.apache.org/docs/. They are there, for the purpose so that you can deploy Airflow in the way you think is best, but if you state no expectations on your side, it's not possible to give you any recommendations.

State your expectations, try to find answers in the docs, then you might attempt to add specific question stating what you want to achieve and what you found so far and especially where you miss answers.