Airflow with virtualenv and private repositories

[stemarks]

Hi All,

We are new to Airflow and have setup some basic DAGs with little effort, however we are now looking at more complex DAGs and have hit an issue with SQL Alchemy dependencies clashing with airflow. We need version 2.0 + as we use SQLModel but airflow uses 1.x so conflicts.

so now we want to use virtual env's, the issue is we have all our python packages live in private github repositories.

We need to be able to run all our dags in their own virtual env, but the packages we use for our tasks are stored in private github repos, we are a GCP house so could put the wheels onto Artifact Registry if that makes it easier.

The biggest thing that im not sure how to get around is the authentication to our github repositories, is there an ENV variable that i can set to tell it to use for authentication when it gets a git+https://github.com in the requiremenets? The documentation i have read doesnt really give a clear example of doing this.

I would also want it to be cached so that it doesnt re-install every time the dag runs unless a version has changed.

I'm sure that this is a common use case but struggling to progress due to lack of knowledge

We are using the official Apache Airflow Helm chart for our deployment
We use a Github App for authentication to git repos (could we use git sync??)

[stemarks]

I have attempted to get a version wor work with virtualenv using @task.external_python and then creating the venv as part of the dockerfile build. but the dockerfile build fails due to conflicts still. I assume it is related to using the postgres hook

@task.external_python(task_id="mitre_check", python="/opt/airflow/mitre-venv/bin/python")
    def callable_mitre_check():
        """Connects to the database and performs a data check."""
        try:
            conn = PostgresHook(postgres_conn_id="my_db_id").get_connection("core_id")
            connection_url = URL.create(
                drivername="postgresql+psycopg2",
                username=conn.login,
                password=conn.password,
                host=conn.host,
                port=conn.port,
                database=conn.schema,
            )
            engine = create_engine(connection_url)
            asyncio.run(run_mitre_check(engine))
        except Exception as e:
            logging.error(f"Error in mitre_check: {e}")
            raise  # Reraise the exception to mark the task as failed

    mitre_check = callable_mitre_check()

Im now stuck short of downgrading our app to use sqlalchemy v1 and stop using SQLModel which we dont really want to do.

[akouirouk]

Having the same problem here. I have looked at quite a few issues relating to this topic, and it seems like the workaround for sqlalchemy is to install apache-airflow using pip and constraint files, but it doesn't address anything about sqlmodel. I feel like this should be looked it, as I'm sure there are other people struggling with this as well.

[potiuk]

Sure. It's being looked at. Airflow 3 will have sqlalchemy limit lifted. The limit is mostly about our dependencies (which we remove as required as part of Airflow 3 effort). If you have other ways @akouirouk how it can be handled, creating a PR to do so us the right way to attempt it.

Nice way to give back to the community of people who work on Airflow in their free time.

Do you think you have a way to address it ?

[stemarks]

@potiuk thanks for the response, for now we are continuing to use our old celery tasks until this is resolved, but without SQLAlchemy v2 we are unable to move over unfortunately. I look forward to Airflow v3 to get over that hurdle :)