Airflow not exporting metrics on statsd port

[mihirbhagwat0]

Apache Airflow version

Other Airflow 2 version

What happened

I have deployed Airflow 2.3.4 on kubernetes.

I have used apache/airflow:2.3.4-python3.8 docker image to build my custom docker image.

Airflow works fine on kubernetes but neither I am getting any metrics on 8125/9125 nor it's listening on the same ports although I have provided statsd enabled configuration in my airflow.cfg file.

Attaching the screenshots of my statsd configuration and netstat command output respectively.

Please help me debug this issue.

Thanks.

What you think should happen instead

No response

How to reproduce

No response

Operating System

apache/airflow:2.3.4-python3.8 docker image

Versions of Apache Airflow Providers

Airflow 2.3.4

Deployment

Other Docker-based deployment

Deployment details

I have created my custom kubernetes manifests for airflow.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[ashb]

netstat output is not relevant as Airflow never listens on the statsd port but sends metrics to it.

It will send UDP metrics to the configured location. What is running behind statsd-service? How did you verify that it's not getting metrics? Do all components/pods have the same airflow.cfg?

[mihirbhagwat0]

@ashb , Thanks for your quick reply.

By mistake I said airflow listens on statsd port. I meant I am not able to see that port open in my pod using netstat command.

statsd pod in running behind the statsd-service. All airflow components have same airflow.cfg file

Please check the attached file for the statsd metrics which I am getting.

[ashb]

I meant I am not able to see that port open in my pod using netstat command.

Outbound UDP traffic won't show up in netstat unless you are very lucky (or possibly never), as there is no persistent connection, it's fire-and-forget

[mihirbhagwat0]

Hi @ashb ,

I resolved the issue.

It was a configuration error at statsd k8s service.

Thanks for your help !

[potiuk]

You can also check how statsd is configured in our Helm Chart - we have a separate statsd exporter running next to airflow that exports statsd published by Airflow. See https://airflow.apache.org/docs/helm-chart/stable/index.html

Generally speaking - switching to our Helm Chart when you use K8S already is Highly Recommended - it has a lot of configurability - and it would be far easier for you to extend it with what you need (if missing) rather than prepare the whole deployment on your own. There are a number of details you might miss (for example when you use Celery Executor you need to run your workers with different signal handling than other components, there are liveness checks implamented, log grooming, pgbouncer configured if you want to use Postgres etc. etc. Starting from Airflow Helm Chart will save you a ton of time to find out that you have those problems and figuring out how to do it.

[mihirbhagwat0]

@potiuk , Thanks for your reply.

I have also deployed a separate statsd exporter as a statsd service. I have refered deployment yamls from the helm chart github repo only.

Could you please let me know what might be the cause of this issue and how can I overcome this issue ?

I have attached the netstat command output from my statsd pod which shows the ports it's listening on.

[potiuk]

No idea, unfortunately. You need to debug it. But I strongly recommend you to use our chart instead.

I looked briefly and cannot see any problem immediately, but I do not even understand what your problem really is in this case and it would take me a lot of time to try it. Instead you cna just use our chart and get it working (or if not we can debug our chart).

Unfortunately, we cannot debug your deplioyment other than quick look in this case. This is the way we try to limit the time needed by us (maintainers) to do what we should do (maintain the project).

Debugging individual issues with custom deployment where there is one that works from the community is not something we can spend a lot of time on. And when you produce a free softwre in your free time, helping to debug other's deployment does not scale. We have 10s of thousands of users and that's why we have helm chart that just works - so that we do not debug individual custom deployments.

So I am afraid that you've chosen the "hard path" rather than go the "easy route", so I think you need to bear the consequences a bit and debug it yourself a bit more (unless you turn back and go the easy path).

You will save a lot of time of yours and others. But if you want to learn - feel free. I am just afraid we won't be able to spend a lot of time on debugging your individual deployment in this case.

[mihirbhagwat0]

Hi @potiuk ,

I resolved the issue.

It was a configuration error at statsd k8s service.

Thanks for your help !