Best practice for webserver liveness probe check

[hanxdatadog]

Description

👋 Dear Airflow community,

Recently we ran some stress tests on Airflow’s asset-based scheduling and noticed that the webserver was frequently restarting due to liveness probe failures. The liveness probe we were using was:

/api/v2/monitor/health

This was based on the guidance from the old health endpoint response:

airflow/airflow-core/src/airflow/api_fastapi/core_api/app.py

Line 85 in 31f0eac

content={"error": "Moved in Airflow 3. Please change config to check `/api/v2/monitor/health`"},

From reading the source code, my understanding is that /api/v2/monitor/health checks the overall health of the metadatabase, scheduler, and triggerer. If there’s any slowdown in retrieving health information from these components, the webserver gets restarted, which makes the UI unavailable. Ideally, we’d like the UI to remain available even if the metadb or scheduler is under heavy load.

What would be the recommended alternative liveness check that doesn’t make the webserver’s health dependent on backend components? I see some options, such as the execution API health endpoint:

airflow/airflow-core/src/airflow/api_fastapi/execution_api/routes/health.py

Line 30 in 31f0eac

return {"status": "healthy"}

I also noticed that the official chart for the API server uses the version endpoint:

airflow/chart/templates/api-server/api-server-deployment.yaml

Line 194 in 31f0eac

path: /api/v2/version

Any suggestions or guidance would be much appreciated 🙏

Use case/motivation

A liveness probe check API end point for webserver that is not dependent on other components

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct