Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue on MacOS: HWLOC or PAPI required if not running on a Linux platform #673

Open
Goon83 opened this issue Nov 27, 2019 · 23 comments
Open

Comments

@Goon83
Copy link

Goon83 commented Nov 27, 2019

Mostly follow the steps listed here : #616
But, no luck on my Mac 10.14.6. After much of pain to install PAPI, remove -lrt and other issues like malloc.h. Code did get compiled but it reports the below error at runtime. I used the ./build.minimal.sh /w PAPI turn on. Any idea to debug or resolve this ?

I also tried to test without PAPI via,

adding

#define DASH__UTIL__TIMER_POSIX

to

/Users/dbin/test/dash-git/dash/include/dash/util/Timer.h

It reports the same error too.

Error:

./ex.11.halo-stencil.mpi

[ 0 ERROR ] [ 627018.374 ] hwinfo.c :362 !!! DART: dart_hwinfo: HWLOC or PAPI required if not running on a Linux platform
[ 0 ERROR ] [ 627018.410 ] unit_locality.c :224 !!! DART: Assertion failed: dart_hwinfo(&hwinfo) -- Expected return value 0
application called MPI_Abort(MPI_COMM_WORLD, -6) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=-6
:
system msg for write_line failure : Bad file descriptor

@Goon83 Goon83 closed this as completed Nov 27, 2019
@Goon83 Goon83 reopened this Nov 27, 2019
@devreal
Copy link
Member

devreal commented Nov 27, 2019

@Goon83 Thanks for bringing this back up. I agree that the argument issued in the previous tickets that mac OS X is not an HPC platform is rather weak and that many people use that platform for development work. I think our main problem is that we currently have no one who is using a mac so it's hard to properly debug and fix these issues. Patches are of course always welcome :)

That being said, you mentioned that you installed PAPI. Could you please confirm that CMake properly finds your PAPI installation? A summary of the enabled packages in printed at the end of the CMake run. Does it list PAPI as found? If not, can you make sure the path to PAPI is set the PAPI_PREFIX CMake option and check in the CMake error log why the detection eventually fails?

@Goon83
Copy link
Author

Goon83 commented Nov 28, 2019

@devreal

I think it finds the PAPI. The ./build.minimal.sh says:
...

-- Searching for PAPI in path
-- Found PAPI: /usr/local/lib/libpapi.so
-- PAPI includes: /usr/local/include
-- PAPI libraries: /usr/local/lib/libpapi.so
....

-- MPI implementation: mpich
-- PAPI enabled
-- MKL not found
-- BLAS enabled

As shown above, the /usr/local/lib/libpapi.so is located for compile.
For some reason, after building executable file, it needs manually to provides the PATH for libpapi.so. Also, since the MacOS prefers libpapi.dylib file instead of .so file, I needs to otool and install_name_tool (here) to replace libpapi.dylib with libpapi.so. Otherwise, it reports below error.
Maybe this step introduces the error.

The error for

16:40:02@dash🍃 ./ex.11.simple-stencil.mpi
dyld: Library not loaded: libpapi.dylib
Referenced from: /Users/dbin/work/soft/dash/build/dash/./ex.11.simple-stencil.mpi
Reason: image not found
Abort trap: 6

@Goon83
Copy link
Author

Goon83 commented Nov 28, 2019

BTW, I also tested with hwinfo (which installed from MacPorts). It also reports the same error.

@devreal
Copy link
Member

devreal commented Nov 28, 2019

The error message from DART that you first reported indicated that PAPI was not found at compile time (it's guarded by ifdefs). The linker error suggests that something is wrong with your PAPI installation. Can you check whether the compile commands in DASH contain the flag -DDART_ENABLE_PAPI? That flag should be set if PAPI is found.

As an alternative, is there an easy way for you to install hwloc? That might be more reliable and certainly provides better information for our use case than PAPI...

@devreal
Copy link
Member

devreal commented Dec 11, 2019

@Goon83 Thanks to @anindex we have a patch that tackles the issue of DASH running on Mac: #675 Could you give that a try?

@devreal
Copy link
Member

devreal commented Dec 11, 2019

Note that with that patch you should not require PAPI or hwloc...

@Goon83
Copy link
Author

Goon83 commented Dec 14, 2019

@devreal @anindex, thanks for the efforts. As I am on international travels recently, will test the patch ASAP.

@Goon83
Copy link
Author

Goon83 commented Jan 13, 2020

Hi @devreal and @anindex,
I tested the new patch and the compile works on my Mac.
Also, I ran all the test/example codes and nothing goes wrong .
Thanks for the great work.

Something wired, however, happens after I tried to compile the example code individually /w the dash-mpic++. Error informations are listed below. Note that using the compile files within the code tree works (as said above). Any hints to remove the error ?

>  ~/opt/dash-0.4.0/bin/dash-mpic++  dash/examples/ex.02.matrix/main.cpp
dash/examples/ex.02.matrix/main.cpp:1:10: fatal error: 'unistd.h' file not found
#include <unistd.h>
         ^~~~~~~~~~
1 error generated.

///I removed the #include <unistd.h> file here
> vim dash/examples/ex.02.matrix/main.cpp

> ~/opt/dash-0.4.0/bin/dash-mpic++  dash/examples/ex.02.matrix/main.cpp
In file included from dash/examples/ex.02.matrix/main.cpp:2:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iostream:38:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/ios:215:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iosfwd:96:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/wchar.h:119:15: fatal error:
      'wchar.h' file not found
#include_next <wchar.h>
              ^~~~~~~~~ 

Bests,
Bin

@devreal
Copy link
Member

devreal commented Jan 14, 2020

@Goon83 Thanks for checking the branch :) Something seems off with your setup.

First, the unistd.h header should be available on Mac OS X (it's part of the C POSIX API afaik). Can you search for it in /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/?

Second, it seems that your compiler has an issue with the C++ iostream header. Can you try to compile this simple check using both your native C++ compiler, mpic++, and dash-mpicxx?

#include <iostream>

int main()
{
  return 0;
}

@Goon83
Copy link
Author

Goon83 commented Jan 14, 2020

Hi devreal,
Thanks for the post to help. I had the two tests below with or without "unistd.h" on both compiler mpicxx and dash-mpicxx. The mpicxx works on both cases but the dash-mpicxx failed. I also can find the "unistd.h" on multple locations of my mac (see the end).

Test 1, without "unistd.h"

#include <iostream>

int main()
{
  return 0;
}

mpicc works but the dash-mpicxx fails.

🍀10:08:42@dash🍃 mpicc test.cpp  -o test
🍀10:09:00@dash🍃 ./test
🍀10:09:09@dash🍃 ~/opt/dash-0.4.0/bin/dash-mpicxx test.cpp  -o test
In file included from test.cpp:1:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iostream:38:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/ios:215:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iosfwd:96:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/wchar.h:119:15: fatal error:
      'wchar.h' file not found
#include_next <wchar.h>
              ^~~~~~~~~
1 error generated.

Test 2: with the unistd.h.

#include <unistd.h>
#include <iostream>

int main()
{
  return 0;
}

mpicc works but the dash-mpicxx fails.

🍀10:29:29@dash🍃 ~/opt/dash-0.4.0/bin/dash-mpicxx test.cpp  -o test
test.cpp:2:10: fatal error: 'unistd.h' file not found
#include <unistd.h>
         ^~~~~~~~~~
1 error generated.
🍀10:29:42@dash🍃 mpicxx  test.cpp  -o test

Found "unistd.h" in a few places.

🍀10:42:29@dash🍃 locate unistd.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/unistd.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/unistd.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/unistd.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/:

@devreal
Copy link
Member

devreal commented Jan 14, 2020

Interesting. Can you post the output of the following?

$ ~/opt/dash-0.4.0/bin/dash-mpicxx -dash:verbose test.cpp  -o test

That will show us how the DASH wrapper invokes the underlying MPI compiler.

@Goon83
Copy link
Author

Goon83 commented Jan 14, 2020


🍀11:17:06@dash🍃  ~/opt/dash-0.4.0/bin/dash-mpicxx -dash:verbose test.cpp  -o test
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -DDASH_RELEASE -I/Users/dbin/opt/dash-0.4.0//include -I/usr/local/Cellar/mpich/3.3.1/include -DDASH -DDASH_ENABLE_TEST_LOGGING -DDASH_HAVE_STD_TRIVIALLY_COPYABLE -DDASH_ENV_HOST_SYSTEM_ID=default -DDASH_MPI_IMPL_ID=mpich --std=c++14 test.cpp -o test -L/Users/dbin/opt/dash-0.4.0//lib -ldash-mpi -ldart-mpi -ldart-base -lpthread /usr/local/Cellar/mpich/3.3.1/lib/libmpi.dylib /usr/local/Cellar/mpich/3.3.1/lib/libpmpi.dylib
test.cpp:2:10: fatal error: 'unistd.h' file not found
#include <unistd.h>
         ^~~~~~~~~~
1 error generated.
🍀11:17:09@dash🍃 mpicc --verbose
Apple clang version 11.0.0 (clang-1100.0.33.8)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch x86_64 -macosx_version_min 10.14.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o a.out -L/usr/local/Cellar/mpich/3.3.1/lib -flat_namespace -commons use_dylibs -lmpi -lpmpi -L/usr/local/lib -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.osx.a

@devreal
Copy link
Member

devreal commented Jan 14, 2020

What happens if you just invoke the Xcode C++ compiler?

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ test.cpp -o test

@Goon83
Copy link
Author

Goon83 commented Jan 14, 2020

 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ test.cpp -o test
test.cpp:2:10: fatal error: 'unistd.h' file not found
#include <unistd.h>
         ^~~~~~~~~~
1 error generated.

@devreal
Copy link
Member

devreal commented Jan 14, 2020

How do you usually invoke the C++ compiler? /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ not your standard compiler?

@Goon83
Copy link
Author

Goon83 commented Jan 14, 2020

Mostly, I just used mpicxx, not directly using c++.

@devreal
Copy link
Member

devreal commented Jan 14, 2020

Mhh, then mpicxx does something that fixes it. You're using MPICH right? Can you post what mpicxx -compile_info prints out?

@Goon83
Copy link
Author

Goon83 commented Jan 14, 2020

🍀12:10:22@dash🍃 mpicxx -compile_info
clang++ -Wl,-flat_namespace -Wl,-commons,use_dylibs -I/usr/local/Cellar/mpich/3.3.1/include -L/usr/local/Cellar/mpich/3.3.1/lib -lmpicxx -lmpi -lpmpi

@devreal
Copy link
Member

devreal commented Jan 14, 2020

Ahh, so mpich picked up clang++ instead of the Xcode c++.

Can you try to rebuild DASH with the following?

$ CC=clang CXX=clang++ bash build.sh

That should tell the DASH CMake to use the Clang compiler.

@Goon83
Copy link
Author

Goon83 commented Jan 14, 2020

Seems that using the clang++ still has the error:

🍀13:46:18@dash🍃 ~/opt/dash-0.4.0/bin/dash-mpicxx  -dash:verbose test.cpp
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DDASH_RELEASE -I/Users/dbin/opt/dash-0.4.0//include -I/usr/local/Cellar/mpich/3.3.1/include -DDASH -DDASH_ENABLE_TEST_LOGGING -DDASH_HAVE_STD_TRIVIALLY_COPYABLE -DDASH_ENV_HOST_SYSTEM_ID=default -DDASH_MPI_IMPL_ID=mpich --std=c++14 test.cpp -L/Users/dbin/opt/dash-0.4.0//lib -ldash-mpi -ldart-mpi -ldart-base -lpthread /usr/local/Cellar/mpich/3.3.1/lib/libmpi.dylib /usr/local/Cellar/mpich/3.3.1/lib/libpmpi.dylib
test.cpp:2:10: fatal error: 'unistd.h' file not found
#include <unistd.h>
         ^~~~~~~~~~
1 error generated.

For reference, output of mpicc

🍀13:46:49@dash🍃 mpicc --verbose test.cpp
Apple clang version 11.0.0 (clang-1100.0.33.8)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.14.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name test.cpp -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fno-strict-return -masm-verbose -munwind-tables -target-sdk-version=10.14 -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -ggnu-pubnames -target-linker-version 512.4 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I /usr/local/Cellar/mpich/3.3.1/include -I/usr/local/include -stdlib=libc++ -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1 -Wno-framework-include-private-from-public -Wno-atimport-in-framework-header -Wno-extra-semi-stmt -Wno-quoted-include-in-framework-header -fdeprecated-macro -fdebug-compilation-dir /Users/dbin/work/soft/dash -ferror-limit 19 -fmessage-length 80 -stack-protector 1 -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fobjc-runtime=macosx-10.14.0 -fcxx-exceptions -fexceptions -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o /var/folders/3s/_7d04bns3v97z45bg0hdwd580000gp/T/test-5c10b7.o -x c++ test.cpp
clang -cc1 version 11.0.0 (clang-1100.0.33.8) default target x86_64-apple-darwin18.7.0
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/Cellar/mpich/3.3.1/include
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch x86_64 -macosx_version_min 10.14.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o a.out -L/usr/local/Cellar/mpich/3.3.1/lib -flat_namespace -commons use_dylibs /var/folders/3s/_7d04bns3v97z45bg0hdwd580000gp/T/test-5c10b7.o -lmpi -lpmpi -L/usr/local/lib -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.osx.a


@devreal
Copy link
Member

devreal commented Jan 16, 2020

AFAICS, your C++ compiler installation is broken as the compiler is unable to compile a simple C++ application by itself. For some weird reason MPICH seems to correct for this by including the necessary directories but that is not the standard (at least on any sane system I have access to).

Having said that, you can try to correct it yourself by adding something like -I/usr/local/include -stdlib=libc++ -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1 (copied from the mpicxx output, not sure that is sufficient). That is not a problem of DASH anymore though. I would recommend making sure that the underlying compiler infrastructure works correctly though.

@Goon83
Copy link
Author

Goon83 commented Feb 27, 2020

@devreal
Looked back to some issues posted before and noticed that you already merged the Mac branch into dev. The Mac branch works on my Mac OS now. Thanks for the work on it.
One minor issue I forget to mention is that it reports an error at compile time (see below).
It is not a serious one and can be easily fixed by adding below line to the header. You may consider to add it to the dev branch too .

#if _LIBCPP_STD_VER < 14

Error info:

/Users/dbin/opt/dash-0.4.0//include/cpp17/cstddef.h:10:12: error: redefinition
      of 'byte'
enum class byte : unsigned char {};
           ^
/Library/Developer/CommandLineTools/usr/include/c++/v1/cstddef:65:12: note:
      previous definition is here
enum class byte : unsigned char {};

@devreal
Copy link
Member

devreal commented Feb 27, 2020

This has been addressed in #677 (pending review) :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants