Using poetry with existing Airflow

[aratnikovmxmgt]

Greetings everyone!

I have a Dockerfile with Airflow

FROM apache/airflow:2.7.1-python3.11
... several COPY ...
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt

And it works like a charm, but I want to use poetry instead of it. And that's the file I've got.

FROM apache/airflow:2.7.1-python3.11
... several COPY ...
RUN pip install poetry
COPY ./pyproject.toml /opt/airflow/pyproject.toml
WORKDIR /opt/airflow/
RUN poetry install --no-interaction --no-ansi

But while build I've got information about Creating virtualenv dwh-sxWGcd-a-py3.11 in /home/airflow/.cache/pypoetry/virtualenvs
If I do poetry config virtualenvs.create false poetry install trying to install everything into the /usr/local/lib/python3.11/site-packages/ instead of /home/airflow/.local/lib/python3.11/site-packages and I've got permission error (seems that it uses system directory)

And my question is: Is it possible to do installation to local folder with poetry without any virtual env?

[clintonroy]

No. Poetry follows the well trodden path of using virtual environments for deployment. Typically for a docker image, people set virtualenvs.in-project to True, then add /home/airflow/.venv/bin to $PATH to activate the the .venv in a docker context.

[aratnikovmxmgt]

So that means, that if I do

RUN pip install poetry
COPY ./pyproject.toml /opt/airflow/pyproject.toml
WORKDIR /opt/airflow/
RUN poetry install --no-interaction --no-ansi

will not work with airflow?

[clintonroy]

It's your pyproject.toml (that I can't see), it's your docker image, try it? I've already given you all the tips to a typical poetry usage of docker.

[abn]

As mentioned by @clintonroy and several times elsewhere, not using a virtual environment is not recommended. Even within docker containers.

For your use case, which I suspect is to make use of already installed platform libraries, you can make use of the virtualenvs.options.system-site-packages option. This gives the virtual environment access to the system site-packages directory.

poetry config virtualenvs.options.system-site-packages true

However, note that whenever you use poetry to manage dependencies, poetry will always prefer the versions that are in your lock file. So if you lock file says apache-airflow==2.8.1 and you container already has apache-airflow=2.7.1, it will be upgraded as this is the whole point of the lock file.

Alternatively, if you are adamant on using no venv, you can do something like this. The above issue still applies.

USER 0
RUN poetry install --no-interaction --no-ansi
USER airflow

If you do not care for the locked versions, then you can do something like this even with a Poetry project.

COPY ./* /opt/airflow/
WORKDIR /opt/airflow/
RUN pip install /opt/airflow