Web server fails to start when GUNICORN_CMD_ARGS environment variable is specified

[mjarosie]

Apache Airflow version

2.4.0

What happened

When I spin up Airflow following docker compose tutorial with GUNICORN_CMD_ARGS environment variable specified for airflow-webserver container - the web server exits with code 1, logging the following errors:

[2022-09-23 18:36:17,423] {cli_action_loggers.py:125} WARNING - Failed to log action with (sqlite3.OperationalError) no such table: log
[SQL: INSERT INTO log (dttm, event, owner, extra) VALUES (?, ?, ?, ?)]
[parameters: ('2022-09-23 18:36:17.421379', 'cli_webserver', 'default', '{"host_name": "de3a5f6ef177", "full_command": "[\'/home/airflow/.local/bin/airflow\', \'webserver\']"}')]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
ERROR: You need to initialize the database. Please run `airflow db init`. Make sure the command is run using Airflow version 2.4.0.

I have definitely initialised the database using docker compose up airflow-init and have tried invoking docker-compose run airflow-cli db init instead.

What you think should happen instead

Web server should start, the same way it does start without issues when GUNICORN_CMD_ARGS environment variable is not provided.

How to reproduce

Steps to reproduce:

# Following instructions from this page: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.4.0/docker-compose.yaml'
mkdir -p ./dags ./logs ./plugins
echo -e "AIRFLOW_UID=$(id -u)" > .env

Then modify settings of airflow-webserver container in line 115 of docker-compose.yaml so that it looks like this (note the added environment field):

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully
    environment:
      GUNICORN_CMD_ARGS: "--log-level DEBUG"

Then run:

docker compose up airflow-init
docker compose up

Before running docker compose up I made sure I don't have leftover volumes, airflow-init is definitely creating everything from scratch:

$ docker volume ls
DRIVER    VOLUME NAME
$

Operating System

Ubuntu 20.04 LTS

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

Running locally

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

This is coming from wrong understanding on how docker-compose and specifically yaml anchors works.

By addding "environment" section you overrode entire "environment" section from the common yaml anchor <<: *airflow-common section and what you get is a default (empty) sqlite database.

You need to find a way to add environment variable rather than override those common ones. You likely need to copy the common part to your webserver and add your environment there. Or you can add the GUNICORN envv in the common parat (it will not hurt if it is also specified for other components)

Note that the docler compose we have is "quick-start" only - it's not suitable to extend and append stuff easily and if you want to make anything even close to something that you could use in production, you have to heavily modify it and possibly even rewrite completely (and I can tell you there many similar traps you can fall in that docker-compose has - unless you know it very deeply).

Highly recommend switching to something more suitable (our Helm Chart and k8S are much more robust)

[mjarosie]

Thanks so much! Putting the environment variable in the common part fixed the issue indeed. As I'm just starting with Airflow, my plan was to play around with it by spinning it up locally using Docker Compose as that's what I'm most familiar with. Thanks for the suggestion, I'll look into running it using minikube, k3s or something similar.

[potiuk]

Docker Compose seems deceptively simple, but it has MANY traps :)