FORBIDDEN access to worker's logs

[denisvolokh]

Apache Airflow version:
Airflow version: 2.1.2

Airflow Web UI is not able to fetch logs (actually stopped working) from the worker's container. It fails with the following error:

*** Log file does not exist: /app/airflow/logs/PROD-synch_strategies_data/t_synch_strategies_summaries/2021-07-27T06:08:44.532400+00:00/1.log
*** Fetching from: http://a32f9e563f01:8793/log/PROD-synch_strategies_data/t_synch_strategies_summaries/2021-07-27T06:08:44.532400+00:00/1.log
*** Failed to fetch log file from worker. 403 Client Error: FORBIDDEN for url: http://a32f9e563f01:8793/log/PROD-synch_strategies_data/t_synch_strategies_summaries/2021-07-27T06:08:44.532400+00:00/1.log
For more information check: https://httpstatuses.com/403

When I checked the logs at the worker container, I can see all logs at the worker's container.

What could be wrong here?

[potiuk]

You need to have Webserver secret key the same in webserver and workers: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key

[denisvolokh]

Thank you, secret key helped to resolve my problem

[mujohiddin]

@denisvolokh Can you tell me how you fixed it?

[mnaranjorion]

Good morning,
I am installing an Airflow cluster with CeleryExecutor. The deployment of the services on the different machines works correctly, and the distribution of tasks when executing a DAG apparently also.
However, I can't see the logs of the tasks executed in the nodes where the WebServer is not present, although the secret_key of the 'airflow.cfg' file is the same for all the nodes.
I have also checked the time and set the same timezone for all machines.

Same error as in the first message from @denisvolokh.

*** Log file does not exist: <path_to_logs>/logs/dag_id=example_branch_labels/run_id=manual__2022-05-09T09:05:11.152836+00:00/task_id=task1/attempt=1.log
*** Fetching from: http://<server_name>:8793/log/dag_id=example_branch_labels/run_id=manual__2022-05-09T09:05:11.152836+00:00/task_id=task1/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://<server_name>:8793/log/dag_id=example_branch_labels/run_id=manual__2022-05-09T09:05:11.152836+00:00/task_id=task1/attempt=1.log'
For more information check: https://httpstatuses.com/403

What configuration could I be missing?

thank you very much,

[potiuk]

Seting TIME not timezone on all machines is important. Make sure time is synchronized on all your machines. Timezone does not matter, but syncing time (with ntp for example) is crucial.

[potiuk]

Also -you might have some firewalls in between your webserver and workers - so you need to track down any networking you have between webserver and worker.

[potiuk]

Another thing is - maybe you somewhat override the secret. There are many ways airflow configuration variables are set - https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html. - variables, secrets, _CMD. Maybe you have different settings outside of teh .cfg - or maybe you are using different .cfg file on one of your webserver/worker altogether.

I saw that happening in the past.

[mnaranjorion]

Good morning,
After fixing a configuration problem with NTP, I have managed to view the log files of workers installed on machines other than the WebServer.

Thank you very much for your help.

[mbhavik91]

@mnaranjorion : Could you please share details on how you fixed configuration problem with NTP?

[manojnkhunte]

firstly login to all airflow containers like scheduler,webserver,worker and see if you see same time . if yes it means time is already synchronised.
then place below variable AIRFLOW__WEBSERVER__SECRET_KEY like below ....
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.10.0}

build: .

environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__WEBSERVER__SECRET_KEY: 'your_airflow_webserver_sec_key'
you can copy paste AIRFLOW__WEBSERVER__SECRET_KEY as it is .
.... it works with this , hope it helps