editable install cannot find pybind built .so

[HernandoR]

error

after pip install -e .
import odometry_py gives
ModuleNotFoundError: No module named 'odometry_py'

expected behaviour

import odometry_py as opy

details

I have such a project that uses pybind11 build cpp codes into a .so file then been called from a python script as follows:

try:
    sys.path.append(os.path.split(Path(__file__).resolve())[0] + "/../build")
    import odometry_py as opy

i'm recently migrating the project to vcpkg and uv managed environment. therefore with following pyproject.toml

[build-system]
requires = ["scikit-build-core>=0.10", "pybind11", "jinja2"]
build-backend = "scikit_build_core.build"

[project]
name = "odometry_py"
version = "0.0.1"
description = "A minimal example package (with pybind11)"
readme = "README.md"
requires-python = ">=3.9, <3.11"
classifiers = [
    "Development Status :: 4 - Beta",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
]
dependencies = [
    "scikit-learn>=1.6.1",
    "scipy>=1.13.1",
]

[tool.scikit-build]
wheel.expand-macos-universal-tags = true
minimum-version = "build-system.requires"
build-dir = "build/{wheel_tag}"
[tool.scikit-build.cmake]
build-type = "Release"
args = ["-GNinja"]
[tool.scikit-build.cmake.define]
CMAKE_CXX_STANDARD = "17"
CMAKE_CXX_FLAGS = "-Wno-deprecated-declarations"
CMAKE_EXPORT_COMPILE_COMMANDS = "ON"
CMAKE_TOOLCHAIN_FILE = "/home/lz/vcpkg/scripts/buildsystems/vcpkg.cmake"
# VCPKG_HOST_TRIPLET = "x64-linux"
# VCPKG_TARGET_TRIPLET = "x64-linux-static"

both uv build and cmake build can build the lib:

build/cp310-cp310-linux_x86_64/odometry_py.cpython-310-x86_64-linux-gnu.so
build/odometry_py.cpython-310-x86_64-linux-gnu.so

uv pip install -e . also build the file and crate corresponding dir in venv

.venv/lib/python3.10/site-packages/_odometry_py_editable.pth
.venv/lib/python3.10/site-packages/_odometry_py_editable.py
.venv/lib/python3.10/site-packages/odometry_py-0.0.1.dist-info/

but in python, import odometry_py as opy throws ModuleNotFoundError: No module named 'odometry_py'

request

if anyone can have a look on what did i missed?

[LecrisUT]

One concern is that the module does not belong to any package. I don't think that path is covered by the tests (still WIP to get all cases covered #906). Just so that we can narrow the issue, try to:

• Create a src/odometry_py/__init__.py (could be something else, just minimizing the changes needed in the pyproject.toml)
• Add to your pyproject.toml

  [tool.scikit-build]
  wheel.install-dir = "odometry_py"

• Rename your python module in the CMakeLists.txt to anything else (common pattern I've seen is to just make it private like _odometry_py, but anything will be fin), and then try to import odometry_py._odometry_py

This would narrow down the issue if this is an issue with top-level modules.

[HernandoR]

One concern is that the module does not belong to any package. I don't think that path is covered by the tests (still WIP to get all cases covered #906). Just so that we can narrow the issue, try to:

• Create a src/odometry_py/__init__.py (could be something else, just minimizing the changes needed in the pyproject.toml)
• Add to your pyproject.toml
  [tool.scikit-build]
  wheel.install-dir = "odometry_py"
• Rename your python module in the CMakeLists.txt to anything else (common pattern I've seen is to just make it private like _odometry_py, but anything will be fin), and then try to import odometry_py._odometry_py

This would narrow down the issue if this is an issue with top-level modules.

I'm afraid it was not working as follows:

Is it because of my project structure? How would it supposed to be?

[LecrisUT]

Can you share your project or at least your CMakeLists.txt, it seems the install part is not configured properly. I also noticed that you might have had a name clash between the python module and compiled one so maybe the python sources is also necessary

[HernandoR]

here is the CMakeList.txt, I'm not sure what i should share of python sources.

cmake_minimum_required(VERSION 3.15)
project(lidar_odometry)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse3 -pthread -fext-numeric-literals -fPIC -fopenmp")
set(CMAKE_BUILD_TYPE "Release")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS} -O3")

set(THIRDPARTY ${CMAKE_SOURCE_DIR}/thirdparty/x86_64)

# find_package(PCL REQUIRED) # pcl is not using inner libs
find_package(PCL CONFIG REQUIRED)
# add pcl_io_ply to PCL_LIBRARIES
list(APPEND PCL_LIBRARIES pcl_io_ply)

list(APPEND CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR}/cmake)
find_package(GLOG REQUIRED)
find_package(CERES REQUIRED)
find_package(GFLAGS REQUIRED)
find_package(GTEST REQUIRED)
find_package(FLANN REQUIRED)
find_package(GFLAGS REQUIRED)
find_package(NLOHMANNJSON REQUIRED)
find_package(EIGEN REQUIRED)

# set(CMAKE_CXX_FLAGS "-Wall -fopenmp")
find_package(OpenMP REQUIRED)
if(OpenMP_FOUND)
    message(STATUS "########## Found openmp ##########") 
    set(CMAKE_C_FLAGS ${CMAKE_C_FLAGS} ${OPENMP_C_FLAGS})
    set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} ${OPENMP_CXX_FLAGS})
else()
    message(FATAL_ERROR "Openmp not found!")
endif()
add_definitions(-DUSE_OPENMP)

set(CMAKE_EXPORT_COMPILE_COMMANDS ON CACHE INTERNAL "") # works

option(BUILD_WITH_GUI "Build gui viewer for lio" OFF)
if(BUILD_WITH_GUI)
    add_definitions(-D WITH_LIO_GUI)
    find_package(Pangolin REQUIRED)
endif()

include_directories(src/)

file(GLOB_RECURSE ALL_LIBRARY_HDRS "include/*.h")
file(GLOB_RECURSE ALL_LIBRARY_SRCS "src/*.cpp" "src/*.hpp" "src/*.h")

set(LIO_CORE_LIB lio_core_lib)
add_library(${LIO_CORE_LIB} SHARED ${ALL_LIBRARY_SRCS})

find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
message("OpenCV_INCLUDE_DIRS:" ${OpenCV_INCLUDE_DIRS})
message("OpenCV_LIBS:" ${OpenCV_LIBS})
message("opencv version:" ${OpenCV_VERSION})


# # add link directories


target_include_directories(${LIO_CORE_LIB}
    PUBLIC
    ${CMAKE_CURRENT_SOURCE_DIR}/include
    ${CMAKE_CURRENT_SOURCE_DIR}
    ${GLOG_INCLUDE_DIR}
    ${CERES_INCLUDE_DIR}
    ${FLANN_INCLUDE_DIR}
    ${EIGEN_INCLUDE_DIR}
    ${GTEST_INCLUDE_DIR}
    ${GFLAGS_INCLUDE_DIR}
    ${NLOHMANNJSON_INCLUDE_DIR}
    ${PCL_INCLUDE_DIRS})

option(GCC_LOW_VERSION "gcc version" OFF)
if(GCC_LOW_VERSION)
    set(DEPENDS_LIBS     
        ${GLOG_LIBRARIES}
        ${CERES_LIBRARIES}
        ${FLANN_LIBRARIES}
        ${PCL_LIBRARIES}
        ${GFLAGS_LIBRARIES}
        ${GTEST_LIBRARIES}
        ${OpenCV_LIBS})
else()
    set(DEPENDS_LIBS     
        ${GLOG_LIBRARIES}
        ${CERES_LIBRARIES}
        ${FLANN_LIBRARIES}
        ${PCL_LIBRARIES}
        ${GFLAGS_LIBRARIES}
        ${GTEST_LIBRARIES}
        ${OpenCV_LIBS}
        -lstdc++fs)
    add_definitions(-Dexpfs)
endif()

if(BUILD_WITH_GUI)
    list(APPEND DEPENDS_LIBS ${Pangolin_LIBRARIES})
endif()

target_link_libraries(${LIO_CORE_LIB} PUBLIC ${DEPENDS_LIBS})

add_subdirectory(apps)



option(BUILD_WITH_PYTHON_WRAPPER "Build python wrapper for lio" ON)
if (BUILD_WITH_PYTHON_WRAPPER)
    find_package(Python3 3.10 EXACT REQUIRED COMPONENTS Interpreter Development)
    find_package(pybind11 CONFIG)
    message("python version:" ${Python3_VERSION})
    message("python stdlib:" ${Python3_STDLIB})
    pybind11_add_module(_odometry_py python_bind/odometry_py_bind.cpp)
    target_link_libraries(_odometry_py PRIVATE pybind11::module ${LIO_CORE_LIB} lio_io liotool ${DEPENDS_LIBS})
endif()

[LecrisUT]

Not sure if you hit a limit, but it cut off at the relevant part that would have had install instructions. Python sources wise the output of tree should do.

[HernandoR]

Not sure if you hit a limit, but it cut off at the relevant part that would have had install instructions. Python sources wise the output of tree should do.

i don't think there is a install instruction in my CMakeList.txt, is it a must?

.
├── apps
├── .clang-format
├── cmake
├── CMakeLists.txt
├── .git
├── .gitignore
├── .gitlab-ci.yml
├── include
├── python_bind
├── README.md
├── run_mapping.sh
├── scripts
├── src
├── thirdparty
├── vcpkg.json
└── .venv

there's no python project strucure in the project, but only .py scripts in the pthon_bind, which act as an entry script:

./python_bind
├── odometry_py_bind.cpp
├── batch_run_lidar_odo_from_clip.py
├── run_lidar_odo_from_clip_postpatch.py
└── test_bind.py

[LecrisUT]

i don't think there is a install instruction in my CMakeList.txt, is it a must?

Yes, it must have one. But that would also break in non-editable pip install. Does normal pip install work? If not it could be a simple "fix" to your initial problem (will still have other design issues later on).

there's no python project strucure in the project, but only .py scripts in the pthon_bind, which act as an entry script:

Entry scripts for the user? If so you still have much to configure. Those files are also the one that require the dependencies in your pyproject.toml?