Missing documentation: how to add DAG files to Docker Airflow

[stromal]

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Hey @stromal - first of all you are absolutely free to improve the documentation of the project. Airlfow has ~ 2200 contributors to the project - mostly people like you who found that things need improvement and ... they did it. So feel free to make PRs to provide improvements. You seem to know how things can be improved, so you are absolutely welcome to do it.

Let me just remind you that Airflow is created by the community, for the community and you, as a user are part of the community and you - equally as anyone else can provide such PRs.

We made it super-easy - you can just click "suggest a change on this page" and the PR will be opened for you automatically - and you can add updates directly to the .rst files that get converted into documentation. I am super happy to help you with leading the PRs to completion.

Wnen it comes to adding files to docker - this is not something that can be done, because it depends very much on how you do deployment. There are plenty of ways to do deployment of Airflow https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html and depending on how you deploy airfow you should add DAGs differently. And you will find there all the prerequisties and expectations from your side to understand when you choose the depli

You need to understand that DAGs are dynamic - and unlike "code" of airflow - you need to be able to add them dynamically. Those can be shared volumes, or git sync or others. You can see it mentioned in our Helm Chart documentation https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html (this is there because Helm chart is the only "customisable" and extendable deployment that we are supporting in the community). And if you look for one, you should start for Helm Chart.

In all other cases (including docker-compose) - you have to make sure that you know your deployment tool (for example in docker-compose you know how to map a local folder to airflow DAGs folder for all components). Actually the "dags" folder that you created is already mapped there - so you can just copy your files there and they should be found in all components that need them in the AIRFLOW_DAG folder where they should be. So I believe your expectation:

2.4.0 version the base docker-compose.yml file should automatically copy in between the container and the local dags folders.

It actually does it. Whatever you copy to the "dags" folder (that you created with mkdir -p ./dags ./logs ./plugins) should become available in all the the containers that need them. And this is a good start for you to see how it works . But if you modify docker-compose file, you need ot know what you do - it's easy to make a mistake there and remove/override stuff (this is a nature of docker-compose). Maybe your problem comes from those modifications? Note that the docker-compose file is just a helper, quick-start that allows people who know docker-compose to develop their own but you likely need to do it in the way that will be maintainable in the future (docker-compose is only for people who can actuallty write and manage their own docker compose - if you want to use "out-of-the-box" solution that works for you that you can just configurem, Helm Chart is the best choice.

Automatically it copies over only 1 the latest adde python file.
And even that file is renamed to "example_python_operator"

This is not how it works, but maybe you can actually explain what you mean ? Maybe you made some modifications that make it works this way ?

It Airflow should be as easy to use as gradio.io, or just having python functions and the execution routing should be controlled by the airflow packages.

Agreee - this is something we plan to do and I am a big proponent of that. Currently we have airflow standalone command and I do agree this should be a "production" quality. We are not yet there but I hope we will soon.

The documentation feels like a huge chaos.

Feel free to improve it in any place you see. As mentioned, it is super easy to do.

If you want a GREAT DOCUMENTATION example in the same topic: https://docs.astronomer.io/astro/create-project <-- This is how the official documentation should look like.

Please - by all means transplant this to the open-source version :). PRs on improving the docs are most-welcome. Maybe it will take you a bit of time to learn it, but you seem to have great ideas and once you ge the hang of it, I think this is absolutely one of the best things you can do to pay back for the free software you get. You have to remember that this is a free software, developed by most people in their free time, so contributing your own time back in echange for the free software you get, is one of the best way to pay back respect to all other people who created it. You can help others to understand it, and as user who just struggled with it - you are likely one of the best people to know where and how people will look for such documentation and you are the best person to contribute it back. I sincerely invite you to do so - and again - happy to review and help you with leading it to completion.

BTW. I converted it to a discussion - because this is what it is.

Looking forward to future PRs of yours @stromal