find_package issues when migrating to conan 2

[maichmueller]

Hi,
inb4: thanks a lot for your work on conan and auto-enabling it with cmake! It is well appreciated.

I am currently migrating a project from conan 1 to conan 2, but running into strange find_package problems.

TLDR:

The cmake run with Conan2 tells me multiple times that something could not be found, then immediately tells me it was found, and eventually fails nonetheless, because it was not found after all:

-- Could NOT find Python3 (missing: Python3_DIR)
-- Found Python3: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter Development NumPy Development.Module Development.Embed
...
-- Could NOT find NumPy (missing: NumPy_DIR)  
-- CMake-Conan: find_package(PythonInterp) found, 'conan install' already ran
-- Could NOT find PythonInterp (missing: PythonInterp_DIR)
-- Found PythonInterp: /usr/bin/python (found version "3.8.10")
-- Found NumPy: version "1.24.4" /home/michael/.local/lib/python3.8/site-packages/numpy/core/include

Long version:

For conan 1 the project was configured with cmake-conan calling conan from within my cmake call and installing all the deps and providing paths:

[generators]
cmake
cmake_find_package
cmake_paths

I am now attempting to use cmake-conan as called from the CLion-Conan plugin with conan 2. That is the conanfile.py is default generated and my deps are in conandata.yml with CMakeDeps as sole generator and toolchains being created in the generate() method of the conanfile.py.

I am running 2 side-by-side cmake calls, one with conan 1 and another with conan 2, to show the differences in the find_package behaviour. The find-package section of my cmake call is:

find_package(xtensor REQUIRED)
set(PYBIND11_FINDPYTHON FALSE)
find_package(Python3 COMPONENTS Interpreter Development NumPy)
find_package(pybind11 REQUIRED)
# install all CPM installed dependencies  (xtensor-python/blas)
include(${DEPENDENCY_DIR}/CPM_deps.cmake)

Conan 1 cmake log (important bits)

conanfile.txt: Generator txt created conanbuildinfo.txt
........
conanfile.txt: Generator cmake_paths created conan_paths.cmake
conanfile.txt: Generator cmake created conanbuildinfo.cmake
conanfile.txt: Aggregating env generators
conanfile.txt: Generated conaninfo.txt
conanfile.txt: Generated graphinfo
-- Conan: Loading conanbuildinfo.cmake
-- Conan: Adjusting output directories
-- Conan: Using cmake targets configuration
........
-- Conan: Adjusting language standard
-- Current conanbuildinfo.cmake directory: /home/michael/GitHub/reinforce/cmake-build-debug
........
-- Conan: Using autogenerated Findrange-v3.cmake
-- Found range-v3: 0.12.0 (found version "0.12.0") 
-- Found Python3: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter Development NumPy Development.Module Development.Embed    <---- Python3 development paths are all found without fail
-- Conan: Using autogenerated Findpybind11.cmake
-- Found pybind11: 2.10.4 (found version "2.10.4") 
........
-- xtensor-python v0.26.1
ERRORMismatch xtensor versions. Found '..' but requires: '0.24.0'
-- Found pybind11 v2.10.4
-- Found PythonInterp: /usr/bin/python (found version "3.8.10") 
-- Found NumPy: version "1.24.4" /home/michael/.local/lib/python3.8/site-packages/numpy/core/include   <---- Numpy is found correctly by the find_package(Python3 COMPONENTS ...NumPy...) call!
-- Found numpy: /home/michael/.local/lib/python3.8/site-packages/numpy/core/include     <---- Numpy is found again by xtensor-python

Conan 2 cmake log (important bits)

-- CMake-Conan: CONAN_GENERATORS_FOLDER=/home/michael/GitHub/reinforce/cmake-build-release/conan/build/Release/generators
-- CMake-Conan: CONANFILE=/home/michael/GitHub/reinforce/conanfile.py
...........
-- CMake-Conan: find_package(Python3) found, 'conan install' already ran
-- Could NOT find Python3 (missing: Python3_DIR)   <---- Python3 cannot be found
-- Found Python3: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter Development NumPy Development.Module Development.Embed     <----- Nevermind, here it is?
-- CMake-Conan: find_package(pybind11) found, 'conan install' already ran
-- Conan: Component target declared 'pybind11::headers'
...........
-- Conan: Component target declared 'pybind11::python2_no_register'
-- Conan: Target declared 'pybind11_all_do_not_use'
-- Conan: Including build module from '/home/michael/.conan2/p/pybin6006d5edca4af/p/lib/cmake/pybind11/pybind11Common.cmake'
...........
-- CMake-Conan: find_package(xtensor-python) found, 'conan install' already ran
-- CPM: Adding package [email protected] (0.26.1)
...........
-- xtensor-python v0.26.1
...........
-- Found pybind11 v2.10.4
-- CMake-Conan: find_package(NumPy) found, 'conan install' already ran
-- Could NOT find NumPy (missing: NumPy_DIR)  <--- Apparently numpy could no longer be found, but further down it easily can
-- CMake-Conan: find_package(PythonInterp) found, 'conan install' already ran
-- Could NOT find PythonInterp (missing: PythonInterp_DIR)   <--- Again python could not be found it
-- Found PythonInterp: /usr/bin/python (found version "3.8.10")      <--- Here I am told it is found...
-- Found NumPy: version "1.24.4" /home/michael/.local/lib/python3.8/site-packages/numpy/core/include  <--- this is xtensor-python's find_package(NumPy REQUIRED) call. This is working fine..., but I am still getting the error of NumPy not found
CMake Error at cmake-build-release/_deps/xtensor-python-src/CMakeLists.txt:58 (find_package):
  By not providing "FindNumPy.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "NumPy", but
  CMake did not find one.

  Could not find a package configuration file provided by "NumPy" with any of
  the following names:

    NumPyConfig.cmake
    numpy-config.cmake

  Add the installation prefix of "NumPy" to CMAKE_PREFIX_PATH or set
  "NumPy_DIR" to a directory containing one of the above files.  If "NumPy"
  provides a separate development package or SDK, be sure it has been
  installed.

Please note, some dependencies (xtensor-python, xtensor-blas) are pulled in by the cmake dependency tool CPM, since they do not have a conan recipe.

Where do these duplicate find_package calls come from?
Any advice on how to make the find_package work in conan 2 as well are well appreciated.

[memsharded]

Hi @maichmueller

Thanks for your report.
It is not fully clear what are the Conan dependencies and what are not. Is NumPy a Conan dependency? What package depends on NumPy?
If it is not a Conan dependency but a CPM one, and it is xtensor (also a CPM dependency), then I am not fully sure how this is connected to Conan?

Please note, some dependencies (xtensor-python, xtensor-blas) are pulled in by the cmake dependency tool CPM, since they do not have a conan recipe.

It would be great to have them in Conan too. I see https://github.com/xtensor-stack/xtensor-python is a header-only library, so it should be quite easy to add it to ConanCenter too. Maybe the challenge is if it has dependencies that are not header-only and are challenging to package.

[maichmueller]

Hi @memsharded ,

sorry for the late reply!
I feared my post may be a little confusing. To clarify: The CPM dependencies are only xtensor-python and xtensor-blas. The former depends on NumPy which is not a Conan dependency (there is no NumPy recipe in Conan right?), and thus on Python3 from which to grab the numpy include dirs. Python3 is also not a Conan dependency but I expect to be installed somewhere in the system for my cmake setup to work.

After investigating my issues with the migration more, I guess my main point is that cmake-conan 2 prints some weird statements that are confusing to an end-user who does not know what Conan is doing under the hood and needs to debug their setup. The confusion came from the contradictory find_package statements which one time tell me NOT FOUND and one time tell me FOUND. I suppose the corresponding conan_provider.cmake lines are these:

    # Filter out `REQUIRED` from the argument list, as the first call may fail
    set(_find_args "${ARGN}")
    list(REMOVE_ITEM _find_args "REQUIRED")
    if(NOT "MODULE" IN_LIST _find_args)
        find_package(${package_name} ${_find_args} BYPASS_PROVIDER PATHS "${CONAN_GENERATORS_FOLDER}" NO_DEFAULT_PATH NO_CMAKE_FIND_ROOT_PATH)
    endif()

    # Invoke find_package a second time - if the first call succeeded,
    # this will simply reuse the result. If not, fall back to CMake default search
    # behaviour, also allowing modules to be searched.
    set(_cmake_module_path_orig "${CMAKE_MODULE_PATH}")
    list(PREPEND CMAKE_MODULE_PATH "${CONAN_GENERATORS_FOLDER}")
    if(NOT ${package_name}_FOUND)
        find_package(${package_name} ${ARGN} BYPASS_PROVIDER)
    endif()

I would argue some more explanatory messages around these consecutive find_package statements would help users debug a failing setup. Perhaps an info message like "Attempting to find package PKG_NAME only within conan generators..." ?
Statements like

-- CMake-Conan: find_package(PythonInterp) found, 'conan install' already ran
-- Could NOT find PythonInterp (missing: PythonInterp_DIR)            <--- python could not be found it
-- Found PythonInterp: /usr/bin/python (found version "3.8.10")      <--- Here I am told it is found...

are not telling a cohesive story. Particularly, if that exact package is causing trouble when used later.
More of a suggestion for future work on cmake-conan though :)

A little off-topic:
Regarding adding such CPM deps to Conan...I did try that first actually. As mentioned before, xtensor-python needs NumPy and Python3 which I did not understand how to bring into a conan-recipe correctly. I also found it difficult to get the main ideas from the conan tutorial in the docs. It provides a lot of detail knowledge at every step that I found mostly distracting when all I wanted to do was get a 101 example guide to correctly package a simple header-only lib. I have this problem with much of the docs (not meant to offend, I love what you guys have done with conan, the docs are merely a weaker point IMO).
This may well be my fault though. I will give it a shot again once I have more free time 👍

[valgur]

Regarding the NumPy dependency, I have an open PR with a working recipe for it on CCI, if that helps: conan-io/conan-center-index#20283
It's currently blocked by the outdated CPython package, but that can probably be substituted with an external version (which might even be preferable).