Python bindings don't build on OSX

[seanmcl]

What happened?

I followed the build instructions, which worked well up to the Python bindings step.

Then build StableHLO with python bindings enabled:

This immediately failed, looking for lld. I did a brew install llm which succeeded, and the build made it much further, but failed with

-- Checking for pybind11 in python path...
-- not found (install via 'pip install pybind11' or set pybind11_DIR)
CMake Error at llvm-build/lib/cmake/mlir/MLIRDetectPythonEnv.cmake:35 (find_package):
  Could not find a package configuration file provided by "pybind11"
  (requested version 2.10) with any of the following names:

    pybind11Config.cmake
    pybind11-config.cmake

  Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
  "pybind11_DIR" to a directory containing one of the above files.  If
  "pybind11" provides a separate development package or SDK, be sure it has
  been installed.
Call Stack (most recent call first):
  CMakeLists.txt:192 (mlir_configure_python_dev_packages)

I tried different things to get the pybind cmake files, but after searching around a bit I gave up. There are some other issues on the pybind github saying others had had similar trouble with cmake.

Steps to reproduce your issue

No response

Version information

$ build/bin/stablehlo-translate --version
LLVM (http://llvm.org/):
  LLVM version 20.0.0
  Optimized build with assertions.

[sdasgup3]

Hi @seanmcl

Can you please let me know what you are getting with the followings:

pip install pybind11 

# if the above does not work try out setting the following env to site-packages including the aforementioned cmake files
export pybind11_DIR=<your path>/lib/python3.11/site-packages

The path is set based on the presence of <your path>/lib/python3.11/site-packages/pybind11/share/cmake/pybind11/pybind11Config.cmake

[seanmcl]

Thanks for looking.

$ pip install pybind11

Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Requirement already satisfied: pybind11 in /opt/homebrew/lib/python3.11/site-packages (2.13.6)

The file is there

$ ls /opt/homebrew/lib/python3.11/site-packages/pybind11/share/cmake/pybind11/pybind11Config.cmake
/opt/homebrew/lib/python3.11/site-packages/pybind11/share/cmake/pybind11/pybind11Config.cmake

This made more progress

# export pybind11_DIR=/opt/homebrew/lib/python3.11/site-packages

# ...build...
CMake Error at llvm-build/lib/cmake/mlir/MLIRDetectPythonEnv.cmake:42 (find_package):
  Could not find a package configuration file provided by "nanobind"
  (requested version 2.4) with any of the following names:

    nanobindConfig.cmake
    nanobind-config.cmake

  Add the installation prefix of "nanobind" to CMAKE_PREFIX_PATH or set
  "nanobind_DIR" to a directory containing one of the above files.  If
  "nanobind" provides a separate development package or SDK, be sure it has
  been installed.
Call Stack (most recent call first):
  CMakeLists.txt:192 (mlir_configure_python_dev_packages)

Fixed with

# export nanobind_DIR=/opt/homebrew/lib/python3.11/site-packages/nanobind
# ... build ...

CMake Error at llvm-build/lib/cmake/mlir/AddMLIRPython.cmake:538 (get_target_property):
  get_target_property() called with non-existent target "MLIRPythonSources".
Call Stack (most recent call first):
  llvm-build/lib/cmake/mlir/AddMLIRPython.cmake:488 (_flatten_mlir_python_targets)
  stablehlo/integrations/python/CMakeLists.txt:161 (add_mlir_python_common_capi_library)

among dozens of other similar errors. Sorry to be needy!

[sdasgup3]

Just to make sure we are using MLIR_ENABLE_BINDINGS_PYTHON=ON while building the project.

build_tools/build_mlir.sh ${PWD}/llvm-project/ ${PWD}/llvm-build

[seanmcl]

Yep!

$ STABLEHLO_ENABLE_BINDINGS_PYTHON=ON ./build_tools/github_actions/ci_build_cmake.sh ${PWD}/llvm-build ${PWD}/build
CMake Deprecation Warning at CMakeLists.txt:34 (cmake_policy):
  The OLD behavior for policy CMP0116 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


Building StableHLO as a standalone project.
-- Using MLIRConfig.cmake in: /Users/seanmcl/workplace/sherloc/stablehlo/llvm-build/lib/cmake/mlir
-- Using LLVMConfig.cmake in: /Volumes/workplace/sherloc/stablehlo/llvm-build/lib/cmake/llvm
-- Building with -fPIC
-- Found python include dirs: /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13
-- Found python libraries:
-- Found numpy v:
-- Using explicit pybind11 cmake directory: /opt/homebrew/lib/python3.11/site-packages/pybind11/share/cmake/pybind11 (-Dpybind11_DIR to change)
-- Found pybind11: /opt/homebrew/lib/python3.11/site-packages/pybind11/include (found version "2.13.6")
-- Found pybind11 v2.13.6: /opt/homebrew/lib/python3.11/site-packages/pybind11/include
-- Python prefix = '', suffix = '', extension = '.cpython-313-darwin.so
-- Using explicit nanobind cmake directory: /opt/homebrew/lib/python3.11/site-packages/nanobind/cmake (-Dnanobind_DIR to change)
-- Found nanobind v2.4.0:
-- Python prefix = '', suffix = '', extension = '.cpython-313-darwin.so
CMake Error at llvm-build/lib/cmake/mlir/AddMLIRPython.cmake:538 (get_target_property):
  get_target_property() called with non-existent target "MLIRPythonSources".
Call Stack (most recent call first):
  llvm-build/lib/cmake/mlir/AddMLIRPython.cmake:488 (_flatten_mlir_python_targets)
  stablehlo/integrations/python/CMakeLists.txt:161 (add_mlir_python_common_capi_library)

[GleasonK]

In terms of building python binding's I'd try to stick as close to the CI as possible:

https://github.com/openxla/stablehlo/blob/main/.github/workflows/buildAndTestCMake.yml#L88-L97

    - name: Build and Test StableHLO (with Python bindings)
      shell: bash
      run: |
        pip install tensorflow-cpu nanobind==2.4
        pip install -r "$LLVM_PROJECT_DIR/mlir/python/requirements.txt"
        ./build_tools/github_actions/ci_build_cmake.sh "$LLVM_BUILD_DIR" "$STABLEHLO_BUILD_DIR"
      env:
          CMAKE_BUILD_TYPE: Release
          STABLEHLO_ENABLE_BINDINGS_PYTHON: ON
          STABLEHLO_ENABLE_PYTHON_TF_TESTS: ON

Make sure env vars are the same and the deps installed are the same version.

To that end, my local build script has:

K_LLVM_PROJ="$HOME/llvm-project"
K_LLVM_BUILD="$HOME/llvm-build"
K_STABLEHLO_PYTHON_BUILD="$HOME/stablehlo-api-build"
K_STABLEHLO_PROJ="$HOME/openxla/stablehlo"

MLIR_ENABLE_BINDINGS_PYTHON=ON ./build_tools/github_actions/ci_build_cmake_llvm.sh $K_LLVM_PROJ $K_LLVM_BUILD
STABLEHLO_ENABLE_BINDINGS_PYTHON=ON ./build_tools/github_actions/ci_build_cmake.sh $K_LLVM_BUILD $K_STABLEHLO_PYTHON_BUILD