Handle Airflow setuptools compatibility

[LvffY]

Problem description

If you're like me and you use Airflow 1.10.x, you probably have hit some recent setuptools incompatibility.

Concretely, if you make the following commands :

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip==20.2.4
pip install --upgrade setuptools
pip install --upgrade --no-cache-dir apache-airflow==1.10.15 -c https://raw.githubusercontent.com/apache/airflow/constraints-1.10.15/constraints-3.6.txt

The last command should fail during Flask-OpenID installation with the message error in Flask-OpenID setup command: use_2to3 is invalid.

Problem reason

After some digging, it's actually related to releases >=58 from setuptools that removes support for 2to3 command (release made on 04 Sep 2021).

Manual fix

Actually, you could easily make this works by changing a little bit of the commands above :

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip==20.2.4
## Here, you just have to force setuptools not being on major 58.
pip install --upgrade 'setuptools<58'
pip install --upgrade --no-cache-dir apache-airflow==1.10.15 -c https://raw.githubusercontent.com/apache/airflow/constraints-1.10.15/constraints-3.6.txt

Now our Airflow installation works like a charm 🎉.

Problem in tox

In my case, my team and I were unable to install Airflow during our CI/CD pipelines because of this incompatibility.
Because, we're using tox, we had to pass from a pretty nice tox.ini file like

[tox]
envlist = py36
requires =
    tox-pip-version

[testenv]
pip_version = pip==20.2.4
deps =
    apache-airflow==1.10.15
    py36: -chttps://raw.githubusercontent.com/apache/airflow/constraints-1.10.15/constraints-3.6.txt
commands =
## This command never happens because our install process fails
    pip show apache-airflow

to a much more hand-made setup like

[tox]
envlist = py36
requires =
    tox-pip-version

[testenv]
pip_version = pip==20.2.4
deps =
commands =
    pip install --upgrade setuptools<58
    pip install --upgrade apache-airflow==1.10.15 -chttps://raw.githubusercontent.com/apache/airflow/constraints-1.10.15/constraints-3.6.txt
## This command finally happens
    pip show apache-airflow

Tox fix

I didn't want my team to have such an ugly setup so to fix this problem in tox, I write a all new tox plugin fo fix the version from setuptools : tox-setuptools-version. With this plugin, we come back to a nice tox.ini file :

[tox]
envlist = py36
requires =
   tox-pip-version
   tox-setuptools-version

[testenv]
pip_version = pip==20.2.4
## This is the option provided by the new plugin
setuptools_version = setuptools<58
deps =
   apache-airflow==1.10.15
   py36: -chttps://raw.githubusercontent.com/apache/airflow/constraints-1.10.15/constraints-3.6.txt
commands =
## This command finally happens because our all new plugin
   pip show apache-airflow

Thanks to this setup, we were able to run our CI/CD process again (and unit test our beloved Airflow DAGs).

I hope you'll find this post useful, please tell me if you find anything better !

[potiuk]

Good one. Glad you wrote it here. We addressed the problem For Airlfow 2 and you do not need to do this any more for any of the Airflow 2 version (I became maintainer of Flask-OpenID and released a new Python3-only version of it). But since airlfow 1.10 is in the "maintenance mode" we have not upgraded the constraints for it. However. I think we should upgrade those just for Airflow 1.10.15 to allow people to migrate to the bridge version without any problems.

I will do it in a moment.

[potiuk]

All right. It shoudl be possible now to install Airflow 1.10.15 with

pip install --upgrade --no-cache-dir apache-airflow==1.10.15 -c https://raw.githubusercontent.com/apache/airflow/constraints-1.10.15/constraints-3.6.txt

with latest setuptools. @LvffY - can you please confirm ?

[LvffY]

@potiuk It works like a charm now :)

However I think this « workaround » is pretty good for those who are running Airflow < 1.10.15, right ?

[potiuk]

Absolutely. Great that we have it

[potiuk]

Thanks for double-checking!