Skip to content

07. Managing Embedded Python Dependencies

Jump to bottom
Shuheng Liu (InterSystems) edited this page Sep 30, 2024 · 5 revisions

For Package Developers

To include Python dependencies in an IPM package, a developer can list all dependencies in a requirements.txt file at the root level of the package (in the same directory as module.xml). The format of requirements.txt should follow PEP 508.

For Package Users

When installing an IPM package, IPM will automatically check for the existence of requirements.txt at the package root and, if found, install all Python packages listed in it using pip.

Flexible Python (IRIS 2024.2+)

In IRIS 2024.2 and later, certain platforms (such as Windows and Linux) support flexible Python runtime, or, as we call it, bring-your-own-python (BYOP). Users can dynamically load a Python dynamic library (.dll, .so) of their own choosing and make use of various syntax features and optimizations in different Python versions.

If you are using this feature on Windows, irispip.exe is no longer shipped with IRIS. This causes IPM 0.7.2 and earlier to be unable to install Python dependencies. It is recommended that you upgrade to IPM 0.7.3 or higher, then manually specify the path to either 1) a Python executable or 2) a pip executable.

  1. If you want to specify the path of a Python executable, you need to run zpm "config set UseStandalonePip 0" and zpm "config set PipCaller </absolute/path/to/python/executable>".
  2. If you want to specify the path of a pip executable, you need to run zpm "config set UseStandalonePip 1" and zpm "config set PipCaller </absolute/path/to/pip/executable>".
  3. After doing either of the above, you can verify the configuration by running zpm "config list".

Precedence of Python/Pip Detection

Up until 0.7.2, IPM uses irispip.exe on Windows and whichever pip is found first in $PATH on Unix systems.

Starting from 0.7.3, the process is more customizable.

  1. If PipCaller is set, it will be used for calling pip install ... or python -m pip install ..., depending on the value of UseStandalonePip.
  2. If PipCaller is missing (by default), IPM detects the proper pip caller in the following order:
    • Where available (typically 2024.2+) - Use PythonRuntimeLibrary in iris.cpf to find the directory containing the Python executable.
    • On Windows only, try to find /bin/irispip.exe.
    • Unless UseStandalonePip is explicitly set to 1, try to find whichever python3/python is first available in $PATH.
    • Unless UseStandalonePip is explicitly set to 0, try to find whichever pip3/pip is first available in $PATH.

Automatic Python Version Detection

Starting from 0.7.3, IPM automatically detects the version of your Python runtime when installing Python dependencies. This means you can specify any pip/Python path using config set PipCaller <pip/python>, which doesn't have to be the same one that's specified as dynamic Python at the IRIS level.

Incompatible Python Dependencies When Switching Python Versions

Certain dependencies, such as numpy, require compiling Python ABI-dependent C code and are not reusable across different Python versions. Additionally, both irispip.exe and IPM always install Python packages to the same directory regardless of Python versions. Therefore, if you install numpy while using Python 3.10 and then switch to Python 3.12, import numpy will fail due to incompatible Python ABI.

This inherently cannot be addressed by IPM itself without some heavy hacking around sys.path. The recommended and surefire way to fix it is to uninstall the Python package completely and reinstall it in the new environment.

Clone this wiki locally