Unable to login because of timezone error

[sandeepkalra]

Apache Airflow version

2.10.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I am new to Airflow. I have a custom docker that i built. The ui fails to load. The classic "Something bad has happened. For security reasons detailed information about the error is not" message shows up. in logs for webserver I see the following error

72.17.0.7 - - [13/Nov/2024:11:26:31 +0000] "POST /login/?next=http%3A%2F%2F192.168.50.197%2Fhome HTTP/1.0" 302 239 "http://192.168.50.197/login/?next=http%3A%2F%2F192.168.50.197%2Fhome" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0"

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/auth.py", line 139, in decorated
    return _has_access(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/auth.py", line 163, in _has_access
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/views.py", line 1134, in index
    return self.render_template(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/views.py", line 768, in render_template
    return super().render_template(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flask_appbuilder/baseviews.py", line 342, in render_template
    return render_template(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flask/templating.py", line 147, in render_template
    return _render(app, template, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flask/templating.py", line 130, in _render
    rv = template.render(context)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.12/dist-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/templates/airflow/dags.html", line 44, in top-level template code
    {% elif curr_ordering_direction == 'asc' and request.args.get('sorting_key') == attribute_name %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/templates/airflow/main.html", line 21, in top-level template code
    {% from 'airflow/_messages.html' import show_message %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
    {% import 'appbuilder/baselib.html' as baselib %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flask_appbuilder/templates/appbuilder/init.html", line 42, in top-level template code
    {% block body %}
  File "/usr/local/lib/python3.12/dist-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 16, in block 'body'
    {% block messages %}
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/templates/airflow/dags.html", line 108, in block 'messages'
    {{ super() }}
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/templates/airflow/main.html", line 69, in block 'messages'
    {% call show_message(category='warning', dismissible=false) %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/templates/airflow/_messages.html", line 25, in template
    {{ caller() }}
  File "/usr/local/lib/python3.12/dist-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/airflow/www/templates/airflow/main.html", line 77, in template
    >{{ macros.datetime_diff_for_humans(scheduler_job.latest_heartbeat) }}</time>.
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/airflow/macros/__init__.py", line 124, in datetime_diff_for_humans
    return pendulum.instance(dt).diff_for_humans(since)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pendulum/datetime.py", line 744, in diff_for_humans
    other = self.now()
            ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pendulum/datetime.py", line 165, in now
    dt = datetime.datetime.now(local_timezone())
                               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pendulum/tz/__init__.py", line 51, in local_timezone
    return get_local_timezone()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pendulum/tz/local_timezone.py", line 33, in get_local_timezone
    tz = _get_system_timezone()
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pendulum/tz/local_timezone.py", line 61, in _get_system_timezone
    return _get_unix_timezone()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pendulum/tz/local_timezone.py", line 179, in _get_unix_timezone
    return Timezone(etctz.replace(" ", "_"))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pendulum/tz/timezone.py", line 65, in __new__
    return super().__new__(cls, key)  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/zoneinfo/_tzpath.py", line 73, in find_tzfile
    _validate_tzfile_path(key)
  File "/usr/lib/python3.12/zoneinfo/_tzpath.py", line 87, in _validate_tzfile_path
    raise ValueError(
ValueError: ZoneInfo keys may not be absolute paths, got: /UTC
172.17.0.7 - - [13/Nov/2024:11:26:31 +0000] "GET /home HTTP/1.0" 500 1588 "http://192.168.50.197/login/?next=http%3A%2F%2F192.168.50.197%2Fhome" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0"
172.17.0.7 - - [13/Nov/2024:11:26:32 +0000] "GET /favicon.ico HTTP/1.0" 404 456 "http://192.168.50.197/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0"

What you think should happen instead?

I have changed default_timezone = America/New_York but this does not help.

How to reproduce

Python version: 3.12.7
Airflow version: 2.10.3

Operating System

sudo docker exec -it orch-0 bash [sudo] password for kadmin: Sorry, try again. [sudo] password for kadmin: root@4aa1bcaa1a07:/gu/data# cat /etc/os-release PRETTY_NAME="Ubuntu 24.10" NAME="Ubuntu" VERSION_ID="24.10" VERSION="24.10 (Oracular Oriole)" VERSION_CODENAME=oracular ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=oracular LOGO=ubuntu-logo

Versions of Apache Airflow Providers

pip freeze |grep apache-airflow-providers

apache-airflow-providers-apache-kafka==1.6.1
apache-airflow-providers-common-compat==1.2.1
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-common-sql==1.19.0
apache-airflow-providers-fab==1.5.0
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-http==4.13.1
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-postgres==5.13.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.9.0
apache-airflow-providers-ssh==3.13.1

Deployment

Other

Deployment details

$ docker version
Client:
Version: 27.3.1
API version: 1.47
Go version: go1.23.1
Git commit: 2.fc41
Built: Tue Sep 24 00:00:00 2024
OS/Arch: linux/amd64
Context: default
HOST OS Version (Fedora):
$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="41 (Forty One)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Forty One)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-05-13

Anything else?

The issue is reproducible every time. I tried changing to older release of airflow, but it persists

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[martintalero]

Same issue here.

---

____ |( )_______ / /________ __
____ /| |_ /__ / / __ / __ _ | /| / /
___ ___ | / _ / _ / _ / / // / |/ |/ /
// |// // // // _/____/|__/
HTTP Request: GET https://apacheairflow.gateway.scarf.sh/scheduler?version=2.10.2&python_version=3.11&platform=Linux&arch=x86_64&database=postgresql&db_version=14.4&executor=CeleryExecutor "HTTP/1.1 200 OK"
/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/module_loading.py:42 DeprecationWarning: The airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG class is deprecated. Please use 'airflow.providers.celery.executors.default_celery.DEFAULT_CELERY_CONFIG'. The celery provider must be >= 3.3.0 for that..
/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py:55 AirflowProviderDeprecationWarning: The celery.CELERY_APP_NAME configuration uses deprecated package name: 'airflow.executors.celery_executor'. Change it to airflow.providers.celery.executors.celery_executor, and update the -app flag in your Celery Health Checks to use airflow.providers.celery.executors.celery_executor.app.
Loaded executor: CeleryExecutor
Starting the scheduler
Processing each file at most -1 times
Launched DagFileProcessorManager with pid: 31
Adopting or resetting orphaned tasks for active dag runs
Configured default timezone UTC
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.11/logging/handlers.py", line 73, in emit
if self.shouldRollover(record):
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/logging/handlers.py", line 196, in shouldRollover
msg = "%s\n" % self.format(record)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/logging/init.py", line 953, in format
return fmt.format(record)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/logging/init.py", line 689, in format
record.asctime = self.formatTime(record, self.datefmt)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/log/timezone_aware.py", line 44, in formatTime
dt = timezone.from_timestamp(record.created, tz="local")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/timezone.py", line 319, in from_timestamp
tz = local_timezone()
^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/timezone.py", line 301, in local_timezone
return pendulum.tz.local_timezone()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/init.py", line 51, in local_timezone
return get_local_timezone()
^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/local_timezone.py", line 33, in get_local_timezone
tz = _get_system_timezone()
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/local_timezone.py", line 61, in _get_system_timezone
return _get_unix_timezone()
^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/local_timezone.py", line 179, in get_unix_timezone
return Timezone(etctz.replace(" ", ""))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/timezone.py", line 65, in new
return super().new(cls, key) # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/zoneinfo/_tzpath.py", line 67, in find_tzfile
_validate_tzfile_path(key)
File "/usr/lib/python3.11/zoneinfo/_tzpath.py", line 81, in _validate_tzfile_path
raise ValueError(
ValueError: ZoneInfo keys may not be absolute paths, got: /UTC
Call stack:
File "/opt/airflow/venv/bin/airflow", line 8, in
sys.exit(main())
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/main.py", line 62, in main
args.func(args)
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command
return func(*args, **kwargs)
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/cli.py", line 115, in wrapper
return f(*args, **kwargs)
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
return func(*args, **kwargs)
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 59, in scheduler
run_command_with_daemon_option(
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
callback()
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 62, in
callback=lambda: _run_scheduler_job(args),
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 48, in _run_scheduler_job
run_job(job=job_runner.job, execute_callable=job_runner._execute)
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/session.py", line 97, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/jobs/job.py", line 421, in run_job
return execute_job(job, execute_callable=execute_callable)
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/jobs/job.py", line 450, in execute_job
ret = execute_callable()
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 980, in _execute
self.processor_agent.start()
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/dag_processing/manager.py", line 172, in start
process.start()
File "/usr/lib/python3.11/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.11/multiprocessing/context.py", line 281, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/dag_processing/manager.py", line 247, in _run_processor_manager
processor_manager.start()
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/dag_processing/manager.py", line 483, in start
self.log.info("Processing files using up to %s processes at a time ", self._parallelism)
Message: 'Processing files using up to %s processes at a time '
Arguments: (2,)

[martintalero]

The issue mentioned above happens with airflow version 2.10.3 and 2.10.2. The error message comes from the scheduler service. The image is a custom one as well.

[martintalero]

Problem is in file: airflow/utils/timezone.py

$ python
Python 3.11.10 (main, Sep 7 2024, 18:35:41) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

from airflow.utils import timezone
import pendulum
s=timezone
s.local_timezone()
Traceback (most recent call last):
File "", line 1, in
File "/opt/airflow/venv/lib/python3.11/site-packages/airflow/utils/timezone.py", line 301, in local_timezone
return pendulum.tz.local_timezone()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/init.py", line 51, in local_timezone
return get_local_timezone()
^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/local_timezone.py", line 33, in get_local_timezone
tz = _get_system_timezone()
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/local_timezone.py", line 61, in _get_system_timezone
return _get_unix_timezone()
^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/local_timezone.py", line 179, in get_unix_timezone
return Timezone(etctz.replace(" ", ""))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/venv/lib/python3.11/site-packages/pendulum/tz/timezone.py", line 65, in new
return super().new(cls, key) # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/zoneinfo/_tzpath.py", line 67, in find_tzfile
_validate_tzfile_path(key)
File "/usr/lib/python3.11/zoneinfo/_tzpath.py", line 81, in _validate_tzfile_path
raise ValueError(
ValueError: ZoneInfo keys may not be absolute paths, got: /UTC

[potiuk]

It looks like in some cases your docker container might be build with double / in /etc/localtime symbolic link. Likely because some timezone information is missing or wrongly configured and Pendulum does not work well with it.

python-babel/babel#990 (comment) contains a workaround in babel for similar issue. Applying it to your dockerfiles should fix the issue for you.l

[sandeepkalra]

Thanks everyone. I got success in fixing and would like to share the complete fix.

There were few things that I did in my Docker container.
[1]

RUN set -eux \
        && rm -f /etc/localtime  \
        && ln -s /usr/share/zoneinfo/UTC /etc/localtime

This was the suggestion above, and made sure that the '/etc/localtime' is not mapped to '/usr/share/zoneinfo//UTC' (note the double //). This in itself did not solve the issue.

[2] I found that even when the correct timezone was mentioned in airflow.conf, the settings were not being passed in tz library and it still ended with /UTC. So, I had to add this line as well.

ENV TZ=America/Chicago