pants prohibits python to use PYTHONPYCACHEPREFIX

[thelittlebug]

Describe the bug
as im using django i need a way to start manage.py without a sandbox. normally it is possible to instruct python via the PYTHONPYCACHEPREFIX environment variable to write the compiled bytecode not besides the source files.
thats not working with pants.
a simple test script

import os
import sys


print(os.environ["PYTHONPYCACHEPREFIX"])
print(sys.pycache_prefix)
print(sys.argv)

if i run this script with the system interpreter the output is as expected

❯ python src/python/tutor/test.py
/home/tlb/work/monorepo/__pycache__
/home/tlb/work/monorepo/__pycache__
['src/python/tutor/test.py']

if i run this script with pants the pycache_prefix doesnt get set in the sandbox

❯ pants run src/python/tutor/test.py:src_prod
/home/tlb/work/monorepo/__pycache__
None
['/tmp/pants-sandbox-vN8QhC/src/python/tutor/test.py']

neither without a sandbox

❯ pants run src/python/tutor:test_dev
/home/tlb/work/monorepo/__pycache__
None
['/home/tlb/work/monorepo/src/python/tutor/test.py']

Pants version
2.24.0a0

OS
arch linux

Additional info
my build file in this dir

# cocomo-bazel

vcs_version(
    name="version",
    generate_to="version.py",
    template='version = "{version}"',
)

python_sources(
    name="src_prod",
    dependencies=[
        ":version",
        "./base",
        "./core",
        "./course",
        "./lib",
        "./page",
        "./templates",
        "src/python/tuhls_core",
        "src/python/tuhls_icons",
        "3rdparty/python:reqs#django-allauth",
        "3rdparty/python:reqs#django-htmx",
        "3rdparty/python:reqs#django-jinja",
        "3rdparty/python:reqs#sorl-thumbnail",
        "3rdparty/python:reqs#django-admin-sortable2",
        "3rdparty/python:reqs#djangoql",
        "3rdparty/python:reqs#django-nested-admin",
        "3rdparty/python:reqs#psycopg",
        "3rdparty/python:reqs#ffmpeg-python",
        "3rdparty/python:reqs#whitenoise",
    ],
)

python_sources(
    name="src_dev",
    dependencies=[
        ":src_prod",
        "3rdparty/python:reqs#django-debug-toolbar",
        "3rdparty/python:reqs#django-extensions",
        "3rdparty/python:reqs#Werkzeug",
    ],
)

python_source(
    name="manage_dev",
    dependencies=[":src_dev"],
    restartable=True,
    source="manage.py",
    run_goal_use_sandbox=False,
)

python_source(
    name="test_dev",
    dependencies=[":src_dev"],
    restartable=True,
    source="test.py",
    run_goal_use_sandbox=False,
)


pex_binary(
    name="manage_dev_pex",
    dependencies=[":src_dev"],
    entry_point="manage.py",
    restartable=True,
    layout="loose",
)

python_source(
    name="manage_prod",
    dependencies=[":src_prod"],
    restartable=True,
    source="manage.py",
)

pex_binary(
    name="manage_prod_pex",
    dependencies=[":src_prod"],
    entry_point="manage.py",
)

[thelittlebug]

if i force that value in my entry script the correct cache directory gets used.
but i do not know if thats a good solution because pants is changing the sys.pycache_prefix and i do not know why and if it introduce any bugs if i change it :)

example manage.py

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""

import os
import sys


def main():
    """Run administrative tasks."""
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "tutor.base.settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == "__main__":
    sys.pycache_prefix = os.environ["PYTHONPYCACHEPREFIX"]
    main()

[benjyw]

Hmm, I don't reproduce (not sure why, I half-expect the behavior you're seeing), but you can try adding execution_mode="venv" and venv_hermetic_scripts=False to the pex_binary.

If that doesn't work, can you provide a tiny repo that exposes the problem?

Also, can you explain why Django requires you to do this? https://github.com/pantsbuild/example-django has examples of running manage.py that didn't require setting the cache location, and we've used Pants extensively with Django at a past company and never needed to mess with that, so I'm curious what problem you're encountering.

[thelittlebug]

thank you for your input and time.

in the example repo you linked in your answer, the manage.py command isn't writing the migrations to the right folder but instead into the sandboxed folder.

❯ pants run helloworld/service/admin/manage.py -- makemigrations
Migrations for 'person':
  /tmp/pants-sandbox-jCUk1Z/./helloworld/person/migrations/0003_person_nickname.py
    - Add field nickname to person

so if i change the target to use the 2 params you mentioned above and run the pex instead of the python source, it writes the migration also to the sandbox.

pex_binary(
    name="manage",
    entry_point="manage.py",
    dependencies=[
        ":lib",
    ],
    execution_mode="venv",
    venv_hermetic_scripts=False,
)

❯ pants run helloworld/service/admin:manage -- makemigrations
10:58:46.35 [INFO] Completed: Building 5 requirements for helloworld.service.admin/manage.pex from the lockfiles/python-default.lock resolve: django-stubs==1.10.1, django<4,>=3.2.13, gunicorn>=20.1.0, requests>=2.25.1, types-reque... (11 characters truncated)
Migrations for 'person':
  /home/tlb/.cache/pex/unzipped_pexes/477b66a90606d49950530ff5acb6d3b9158c583c/helloworld/person/migrations/0003_person_nickname.py
    - Add field nickname to person

now the next idea is to run pants source instead of pants pex but without the sandbox. thats the only way i have found to get the output of the makemigrations management command to write into the right folder.

python_source(
    name="manage_from_src_without_sandbox",
    dependencies=[
        ":lib",
    ],
    source="manage.py",
    run_goal_use_sandbox=False,
)

now it runs as intended as you can see on the following output.

❯ pants run helloworld/service/admin:manage_from_src_without_sandbox -- makemigrations
11:04:40.94 [INFO] Completed: Building 5 requirements for manage_from_src_without_sandbox.pex from the lockfiles/python-default.lock resolve: django-stubs==1.10.1, django<4,>=3.2.13, gunicorn>=20.1.0, requests>=2.25.1, types-reque... (11 characters truncated)
Migrations for 'person':
  helloworld/person/migrations/0003_person_nickname.py
    - Add field nickname to person

but it produces everywhere the __pycache__ directories.
i just wanted to collect them somewhere out of my source tree, the possibility for that is builtin in into python since 3.8

that has mainly 2 reasons:

1. i hate them laying around. quickfix would be a wrapper with something like find . -type d -name "__pycache__" -exec rm -r {} +
2. i had issues in the past with the __pycache__ files if theres something fundamentaly changing like the python interpreter version. for sure, the bundled pex file would run as intended because the __pycache__ files are excluded via the .gitignore file. but the direct call via pants run to a python_source target not.

so my conclusion is the following:

1. maybe i'm using the pex wrong?
2. in my testing the pex is not able to write to the right folder only the python_source target with run_goal_use_sandbox=False is.
3. maybe i'm doing something completely wrong than sorry for that noise ^^ but maybe you could give me a hint into the right direction.