Move apache-airflow out of python package set

[mweinelt]

Describe the bug

Apache airflow is an application, the package set is intended for libraries.

Expected behavior

The application should live elsewhere in the repo tree and use buildPythonApplication so it can rely on packageOverrides to get the package version it requires.

Notify maintainers

@ingenieroariel
@bhipple
@gbpdt

[graham33]

The problem with airflow is that some users will use it as an application, and some will also use it as a package (you can write tasks and other scripts that live outside of airflow that need to import the airflow module to interact with the airflow service).

However, clearly it's not possible to support airflow in python3Packages right now since it doesn't support sqlalchemy 2.x. Hence, as mentioned in #214565, I'd still propose to mark it as broken in python3Packages (since hopefully soon apache/airflow#28723 will be addressed and we'll be able to support it again), and keep a working version of the standalone application with its own overridden package set (so people who just need the application can continue using it).

I've implemented this in a draft PR (which works but will need some cleanup): #231944. Again, not ideal but I think better than removing it from the python package set.

[mweinelt]

There is no harm to living outside of the python package set. You inherit all of its packages. Airflow specific packages would need to live in apache-airflow.python.pkgs, which can carry overrides that make it work.

Airflow can use buildPythonApplication and have apache-airflow = toPythonLibrary apache-airflow { }; in its python package overrides.

I don't get the why it needs to live in python packages. It will break again and again with no means to fixing it.

The whole story is similar to ansible (blocking resolvelib updates) and buildbot (being consistently broken, because the upstream releases cannot keep up with our package set update cycle).

#225195 is when buildbot moved out of the package set.

[gbpdt]

For our use-cases there is a benefit to it staying in the python package set: we have some cases where we have a large curated python env, based on python3Packages, which contains airflow.

If airflow is in nixpkgs python3Packages it's obviously less work for us, but more importantly it's tested with the rest of the package set. This means issues like the one we're currently workingon are likely to be found earlier and hopefully resolved earlier. If we add it to python3Packages ourselves in an overlay, we only find the issues when we upgrade nixpkgs and any fixes/forks are likely to remain internal rather than being available to others.

[mweinelt]

Resolved in 08126bf