How to handle __pycache__ in dags folder?

[mertozb]

When i changed some of my config in dag like owner or dag_id Airflow webserver tries to import from pycache in DAG's folder. However, It needs to depend on the original dag.py . Not sure but seeing the logs, i think it may happens when it can't import the original file, this happens like 4 of 20 catchup runs, i dont know if theres some threshold limitations on that like if cant import at x second try the cached dags. Now i fixed it with cleaning the cache but not sure if this happens gradually of increasing use of Airflow 2.4

[Taragolis]

1. Airflow Webserver do not use dag folder at all since Airflow 2, see: https://airflow.apache.org/docs/apache-airflow/stable/dag-serialization.html
2. __pycache__ it is internal of Python, see: https://peps.python.org/pep-3147/ . Python should recompile byte-codes files (.pyc) in case of changing in .py files.

[Taragolis]

Really? I do not have any problem (but I do not exclude possibility that someone may have) with bytecodes except:

1. Need to send to someone folder with code as result it contain this files which might be useless on other system (different Python version, OS).
2. Need to build some distribution like OS package or AWS Lambda

First case not a problem anymore since Python 3.8. I just set PYTHONPYCACHEPREFIX to something like /tmp/pycache
For second case when I need run build script I've execute python with -B flag (same as set PYTHONDONOTWRITEBYTECODE)

[potiuk]

Yes - if you are just running your python code all is fine and it just works. But there are a numer of scenarios where it causes problems.

For developing and switching environments often it is highly problematic (that's why we have it set in all Breeze scripts/env/docker).
On Linux particularly - there are a lot of problems when you develop code and change branches and Python version often for the same codebase, and especially when you have mounted folders to within containers. Usually mounted coee and bytecode are created using root user inside container. And they cannot be deleted if you are not careful and kill your container forecefully.

Also when you change python version that might lead to a number of problems (and this is especially nasty if you have different version in your host and different version of python in the container where your source code is mounted). Then - some files can get bytecode from one version and some from other.

In airflow when DAG files are synced over shared folders and time of the file might get off even by a few seconds, it might happen that .pyc file gets created on one system with earlier time than the file when there are various races and time drifts. and then the problem is that you might get .pyc dated before the .py file even if the file was changed (been there done that).

[Taragolis]

I agree that most of annoying if you incidentally run docker image which internally work as a root with bind mount volume to the host, then you need to root on host for delete __pycache__.

Also set PYTHONPYCACHEPREFIX inside of the docker (very rarely where I have python 3.7, I thought only airflow-ci image) in this case I can be sure that each container:

• Has access to their own bytecodes
• Do not write anything inside of shared volumes and do not create addition IO request to NFS

But might be most easy option in general is set PYTHONDONTWRITEBYTECODE env var

[mertozb]

It seems after the export PYTHONDONTWRITEBYTECODE=1 command, no longer pycache appears. But still scheduler tries to take old dag in some runs.

[mertozb]

Could it be from the paramater:
min_serialized_dag_fetch_interval: This option controls how often the Serialized DAG will be re-fetched from the DB when it is already loaded in the DagBag in the Webserver. Setting this higher will reduce load on the DB, but at the expense of displaying a possibly stale cached version of the DAG.
as the DAG serialization doc stated: https://airflow.apache.org/docs/apache-airflow/stable/dag-serialization.html#dag-serialization-settings