ThreadsafetyTest: Problems with OpenMPI

[devreal]

I am seeing issues on my local machine when running the ThreadsafetyTests under OpenMPI. The used compiler is GCC 5.3.0, both in Debug and Release mode.

OpenMPI 1.10.5
I am experiencing hangs in ConcurrentPutGet, where one or multiple threads are stuck in MPI_Win_flush called by dart_put_blocking:

Thread 4 (Thread 0x7fffe27a8700 (LWP 27525)):
#0  0x00007ffff62b1c47 in sched_yield () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007fffebdfb715 in ompi_osc_pt2pt_flush_lock () from /home/joseph/opt/openmpi-1.10.5/lib/openmpi/mca_osc_pt2pt.so
#2  0x00007ffff752ebe4 in PMPI_Win_flush () from /home/joseph/opt/openmpi-1.10.5/lib/libmpi.so.12
#3  0x000000000095c23b in dart_put_blocking (gptr=..., src=0x7fffe27a7c44, nelem=1, dtype=DART_TYPE_INT) at /home/joseph/src/dash/dash/dart-impl/mpi/src/dart_communication.c:759
#4  0x00000000007cc2a3 in dash::GlobRef<int>::operator= (this=0x7fffe27a7cf0, val=2) at /home/joseph/src/dash/dash/dash/include/dash/GlobRef.h:201
#5  0x000000000089ae16 in dash::GlobRef<int>::operator= (this=0x7fffe27a7cf0, other=...) at /home/joseph/src/dash/dash/dash/include/dash/GlobRef.h:112
#6  0x0000000000898731 in ThreadsafetyTest_ConcurrentPutGet_Test::TestBody () at /home/joseph/src/dash/dash/dash/test/ThreadsafetyTest.cc:74
#7  0x00007ffff67bee46 in gomp_thread_start (xdata=<optimized out>) at ../../../src/libgomp/team.c:119
#8  0x00007ffff79b96ba in start_thread (arg=0x7fffe27a8700) at pthread_create.c:333
#9  0x00007ffff62ce82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

For some reason, the test seems to pass on CircleCI. It seems to be a progress issue and I seem to remember that @fuchsto had investigated something along those lines in OpenMPI. Did you file a bug report back then?

OpenMPI 2.0.2
The test suite does not run at all since OpenMPI 2.0.2 complains about missing support for MPI_THREAD_MULTIPLE in the OSC pt2pt component:

--------------------------------------------------------------------------
The OSC pt2pt component does not support MPI_THREAD_MULTIPLE in this release.
Workarounds are to run on a single node, or to use a system with an RDMA
capable network such as Infiniband.
--------------------------------------------------------------------------
[beryl:10384] *** An error occurred in MPI_Win_create
[beryl:10384] *** reported by process [2174091265,0]
[beryl:10384] *** on communicator MPI COMMUNICATOR 3 DUP FROM 0
[beryl:10384] *** MPI_ERR_WIN: invalid window
[beryl:10384] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[beryl:10384] ***    and potentially your MPI job)
[beryl:10362] 3 more processes have sent help message help-osc-pt2pt.txt / mpi-thread-multiple-not-supported
[beryl:10362] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[beryl:10362] 3 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal

This sounds like a bug in OpenMPI and I will send a report to the user mailing list to ask for clarification. The suggested workarounds are not very helpful (yes, I am running on a single node and no, I do not have Infiniband available on my laptop...) Without any investigation done so far, it's not clear to me though why the openmpi2 test succeeds in CircleCI.

MPICH 3.2
Works as expected 👍

[fuchsto]

This defect only has effect in builds with DASH_ENABLE_THREADSUPPORT, right?
So previously existing functionality is not affected? We could live with that.

[devreal]

The second problem (the failing MPI_Win_create) is under investigation after I posted a report on the user ML. The first problem, though, seems more interesting. I also posted this on the ML and got as a first reply that concurrently calling MPI_Win_flush (which we do in dart_put_blocking to ensure remote completion) is not allowed by the standard. The reply suggested to use MPI_Rput, which does not ensure remote completion, though. The standard does not seem clear on this although you could argue that it's a concurrent operation on a shared (window) object. I asked for clarification.

Depending on the outcome, we might have to ensure mutual exclusion for sync calls ourselves or rethink our synchronization model. Calling MPI_Win_flush after every access surely does not seem to be efficient anyway so we could think about requiring an explicit call to ensure remote completion. OpenMP offers the pragma omp flush directive for example.

So in essence: Yes, this problem currently only bites us if DASH_ENABLE_THREADSUPPORT is enabled but it's certainly an issue in the long run. I will use this issue to track progress.

[devreal]

I got feedback from the OpenMPI folks:

Nathan and I discussed at the MPI Forum last week. I argued that your usage is not erroneous, although certain pathological cases (likely concocted) can lead to nasty behavior. He indicated that he would remove the error check, but it may require further discussion/debate with others.

Sounds like the next release of OpenMPI will support our use-case. We might want to add this (and the broken alignment of shared windows, see #280) to a KNOWN_ISSUES file together with the releases that contain the fixes (as soon as they are available). Other issues (such as missing support for is_trivially_copyable in the Cray compiler, see #241) could go in there as well. Thoughts?

[devreal]

Ping: any opinion on the KNOWN_ISSUES proposal?

[fuchsto]

@devreal I'd be fine with that. Known issues are to be documented in CHANGELOG.md.

[ddiefenthaler]

OpenMPI 2.0.2
The test suite does not run at all since OpenMPI 2.0.2 complains about missing support for MPI_THREAD_MULTIPLE in the OSC pt2pt component:

The OSC pt2pt component does not support MPI_THREAD_MULTIPLE in this release.
Workarounds are to run on a single node, or to use a system with an RDMA
capable network such as Infiniband.
--------------------------------------------------------------------------```

Just as a note, I think that is a problem with the way the version got build.
I had the same error with the package from the system repoistory, but
after building OpenMPI 2.0.2 from sources the error was gone for me.

[devreal]

Interesting. What flags did you pass to configure? I usually build all OpenMPI versions myself.

[ddiefenthaler]

I passed no extra options to ./configure.
I guess there is the problem as it seems I have built without ThreadSupport...
For some reason I also deactivated the thread support in the DASH build.sh...

Sorry my bad, the error persists for OpenMPI 2.0.2. Should've read the output more carefully.

For at least in OpenMPI 2.1.0 this error is gone, but then again it blocks in ThreadsafetyTest.ConcurrentAlgorithm during PMPI_Win_allocate_shared:
Side by side gdb backtrace

[devreal]

That is interesting as well, I was not aware that things changed there. I will follow up with the OpenMPI folks on that. Thanks for the report!

[devreal]

Mhh, this test is failing for me as well and I've been trying to debug this for a while now. The results are wrong and it tends to hang, just as you observe it. However, I get wrong results with MPICH as well, so I assume that we do something nasty there. I just haven't figured out what exactly is going wrong...