ElasticSearch Logs not working

[serrovsky]

Apache Airflow version

2.2.4

What happened

I'm set up my airflow logs to be stored on elasticsearch however when I get to the DAG execution I can't see the log (message below)

*** Log BE.vehicle_sync_get_sync_status_2022_05_16T17_06_09_986757_1 not found in Elasticsearch. If your task started recently, please wait a moment and reload this page. Otherwise, the logs for this task instance may have been removed.

Not sure what's happening, I can't see any error (not from airflow or elasticsearch side...) and I'm not sure how can I debug it as well..

Can anyone give me some help with that?

What you think should happen instead

No response

How to reproduce

No response

Operating System

rbac = True(airflow)cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==3.0.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-cncf-kubernetes==3.0.2
apache-airflow-providers-docker==2.4.1
apache-airflow-providers-elasticsearch==2.2.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.4.0
apache-airflow-providers-grpc==2.0.1
apache-airflow-providers-hashicorp==2.1.1
apache-airflow-providers-http==2.0.3
apache-airflow-providers-imap==2.2.0
apache-airflow-providers-microsoft-azure==3.6.0
apache-airflow-providers-mysql==2.2.0
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==3.0.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sendgrid==2.0.1
apache-airflow-providers-sftp==2.4.1
apache-airflow-providers-slack==4.2.0
apache-airflow-providers-sqlite==2.1.0
apache-airflow-providers-ssh==2.4.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[dhananjayn0712]

Do we have found any resolution for above issue? I am facing same issue on my development instance.

[dhananjayn0712]

Do we have found any resolution for above issue? I am facing same issue on my development instance.

[potiuk]

Well. this is a discussion, because there is not enough details to treat it as an issue and even triage if it is a real issue. So if you have more to add to provide evidences, logs, and providing a reproducible case (for example by opening and issue with all those details) , then there is a chanve it will be looked at by someone. but if not, it's not likely.

BTW. What you see might be a different issue - this discussion is very light on details, so even if you think it is the same issue, it might be not, in which case it is even more appropriate that you open a new issue providing all evidences, logs, reproduction steps and other details that might help in triaging and diagnosing that issue.

Which I heartily recommend you do @dhananjayn0712 .

[y0zg]

I had a similar issue and it was related that log forwarded wasn't properly formated json, so ensure that you have really json airflow logs in elasticsearch

[julienlagorsse-loreal]

We have the same bug, connection seems to be fine in webserver as we get 200 on some count request however we get this in webserver log page
*** Log debug_airflow_to_es-print_debug_message-manual__2024-12-06T12:23:30.109158+00:00--1-1 not found in Elasticsearch. If your task started recently, please wait a moment and reload this page. Otherwise, the logs for this task instance may have been removed.

When I check on elasticsearch there is no index.

The triggerer seems to create a wrapper to redirect to stdout to elasticsearch, we don't really understand the NOTSET part

2024-12-06 13:26:10 [2024-12-06T12:26:10.912+0000] {triggerer_job_runner.py:181} INFO - Setting up TriggererHandlerWrapper with handler <ElasticsearchTaskHandler (NOTSET)>
2024-12-06 13:26:10 [2024-12-06T12:26:10.914+0000] {triggerer_job_runner.py:237} INFO - Setting up logging queue listener with handlers [<RedirectStdHandler (NOTSET)>, <TriggererHandlerWrapper (NOTSET)>]
2024-12-06 13:26:10 [2024-12-06T12:26:10.920+0000] {triggerer_job_runner.py:338} INFO - Starting the triggerer

We don't really use this triggerer for now so it doesn't help much

We can see Listening at: http://[::]:8794 for triggerer

For Worker we don't see a wrapper logs, but we see the listening http://[::]:8793
but nothing else seems to be really related to logging.

I don't find new logs on /opt/airflow/logs, scheduler logs are there but nothing special in it (I have set a cron every min to my test dag)

We aren't really if it's the responsibility of the worker to push logs on elasticsearch, and we don't know who is responsible to write logs.

Elasticsearch is hosted in a docker compose, same network, curl elasticsearch:9200 works everywhere (worker, webserver, ...)

AIRFLOW_CONN_ELASTICSEARCH_DEFAULT: "http://elasticsearch:9200"
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-apache-airflow-providers-elasticsearch}

[logging]
logging_level = INFO
remote_logging = True
remote_log_conn_id=elasticsearch_default
[elasticsearch]
host = http://elasticsearch:9200
write_stdout = True
json_format = True
[elasticsearch_configs]
verify_certs=False