gitsync requirements.txt - install packages

[IOO923]

Is gitsyncing requirements.txt possible? I can't see documentation.

Similiar question posed for older version of values.yml compared to latest.

[potiuk]

No. This is not how python package installation works and how containers work. If you want to use airlfow containers and add custom packages you must build your own image with those packages. This is the only reasonable and supported way for containers: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#extending-and-customizing-airflow-image and you need to have your pipeline to build and publish your image in your registry:

Adding packages to your image with requirements.txt you have a good example here
https://airflow.apache.org/docs/docker-stack/build.html#example-of-adding-packages-from-requirements-txt

You can dynamically create virtulenvs for individual tasks by utilising PythonVirtualenvOpertor if you want though, but changing "system-wide" airflow Python environment in container world must be done with customized image.

[qkponton]

Hi @potiuk
I've seen that it's possible to add a post command to gitsync. so basically post with command "pip install -r requirements.txt" we can manage python dependencies with gitsync.
The side effect of this is that no lock seems to exist between gitsync process and scheduler DAG folder refresh process.
Do you confirm that?
Is it possible to have both process started at the same time and that scheduler refreshs an non up to date folder?

[potiuk]

No . Can't confirm. This is some custom way of using it, if you want to explore it and see if it can work for you -cool. But this is nothing I could personally try or recommend because I have never tried it and there might be non-obvious problems.

But if you battle-test it and have it works - it would be awesome if you write a blog post about it and contribute it to https://medium.com/apache-airflow - that would be the. best way of sharing it with the community and giving back.

[qkponton]

Sorry for the late reply
I was able to test and the requirements were installed by the hook command.
the only thing that wasn't cool is that gitsync update the symlink before running the command.
So it might failed for few seconds until the scheduler refresh

[potiuk]

Cool. As I said, this is a rather custom approach that I would not recommend.

My recommendation is to always build your image with new requirements and deploy them with your change.

There are plenty non-obvious things that might happen in if for example your requirements file gets broken it might end up in -half-installed state, or you can accidentally upgrade airflow by pushing a dependency that will have a conflicting requirements, pip might choose to upgrade airflow for you without you even knowing about it. Or you can have whoever updates the requirements.txt fall into typosquatting vulnerability https://medium.com/@williambengtson/python-typosquatting-for-fun-not-profit-99869579c35d and you might get malicious package installed that will start stealing your secrets, or the package you try to install will pull 100 other packages which will suddenly start breaking everything, including airflow core. Also you can give the user controlling the requirements.txt additional priviledges unintentionally - for example airflow uses entrypoints to determine if plugin interface should be used to load Timetables and few other things that scheduler uses. Those packages, might start running - unintentionally - in scheduler where the code run should not be controlled by the user who has DAG write access. By dynamicaly installing dependencies from DAGs folder you are breaking a basic Airflow security model, which assumes that the "core" airflow environment and which packages are installed there is controlled by Deployment Manager, not by DAG author (DAG author can control what is the virutalenv used for particular tasks, but no more).

There are all kinds of things that you are likely to hit and security vulnerabillities you are opening to when you are allowing to dynamically change the set of dependencies "core" airflow uses. You can likely use it for some kind of isolated development environment but if you are using it for your production deployment, then you should think twice who and how controls the requirements.txt file.

If it works for you cool, if you can write a blog post what kind of limitations it has and problems it introduces - even cooler. But bear in mind this one inherently breaks airflow security model.