Adapting to the system hwloc situation

[inducer]

It seems that we have two possible situations that can give us the annoying:

Choose platform:
[0] <pyopencl.Platform 'Portable Computing Language' at 0x7f0979c86008>
Choice [0]:
Not enough memory to run on this device.
[1]    150038 abort (core dumped)  python wave-eager.py

• System MPI uses libhwloc 1 (Livermore), pocl uses libhwloc 2
• System MPI uses libhwloc 2 (Debian, e.g. Andreas's machine, scicomp cluster), pocl uses libhwloc1

As of #71, we default to installing libhwloc 1, but that's also not a safe default. Is there something we can do to automate installing the correct libhwloc?

This snippet will get the current hwloc version, using either Python 2 or 3:

import ctypes
hwloc = ctypes.cdll.LoadLibrary("libhwloc.so.15")
# https://github.com/open-mpi/hwloc/blob/master/include/hwloc.h
hwloc.hwloc_get_api_version.restype = ctypes.c_uint
hwloc.hwloc_get_api_version.argtypes = []
print(hwloc.hwloc_get_api_version() >> 16)

I'm just not sure that using this is a great idea...

cc @matthiasdiener

[inducer]

I just hit this again. @matthiasdiener, what do you think of the detection solution I proposed?

[matthiasdiener]

See illinois-ceesd/mirgecom#169 for another workaround.

[inducer]

Hm. I just saw this again, but I realized that my system does not benefit from illinois-ceesd/mirgecom#169. Among other things, this breaks running the meshmode tests. I don't think it's feasible to apply this workaround universally. This makes me like the hwloc version picking hackery idea more.

@matthiasdiener What do you think?

[majosm]

illinois-ceesd/mirgecom#306

[inducer]

Based on @majosm's experience, this has the potential to cause spurious breakage on Macs and Linux, except the Mac failures are even worse: all you get is a segfault, not even an error message you can search. I don't think I'd consider this fixed, even if the import order can be used to work around it. (@majosm, could you conda install libhwloc=2 and see if the import order thing even also applies on Macs?)

IMO, the emirge install script should make an effort to set things up so as to avoid this.

The main wrinkle with the script snippet above is that we need some way of finding the hwloc shared library.

[majosm]

(@majosm, could you conda install libhwloc=2 and see if the import order thing even also applies on Macs?)

With conda install libhwloc=2, I get a segfault if I load pyopencl then mpi4py, but not if I load mpi4py then pyopencl.

[inducer]

Thanks for checking! So it's probably the same bug, just different crashes. I don't think it's plausible that we'll catch all the code where things get installed in the "wrong" order. And I can't reasonably add a warning to PyOpenCL either...

@majosm Does the above snippet (or some version of it) successfully detect the "ambient" (MPI) version of hwloc?

[majosm]

@majosm Does the above snippet (or some version of it) successfully detect the "ambient" (MPI) version of hwloc?

Seems like it detects the conda hwloc. When I run:

import ctypes
hwloc = ctypes.cdll.LoadLibrary("libhwloc.dylib")
# https://github.com/open-mpi/hwloc/blob/master/include/hwloc.h
hwloc.hwloc_get_api_version.restype = ctypes.c_uint
hwloc.hwloc_get_api_version.argtypes = []
print(hwloc.hwloc_get_api_version() >> 16)

it prints 2. After conda install libhwloc=1, it prints 1.

[inducer]

But what does it do prior to any conda env being active? The whole point would be for it to detect what hwloc exists in the environment and adapt to that, to avoid conflicts.

[majosm]

After activating conda and prior to conda activate <env-name>, I still get 2. If I don't activate conda at all I get

  File "find_hwloc.py", line 2, in <module>
    hwloc = ctypes.cdll.LoadLibrary("libhwloc.dylib")
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 444, in LoadLibrary
    return self._dlltype(name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(libhwloc.dylib, 6): no suitable image found.  Did find:
	file system relative paths not allowed in hardened programs

[inducer]

Huh. I'm guessing that's the system Python not allowing you to load dynamic libraries? That's annoying.