Radical executor hangs in test_mpi_funcs.py

[matthewc2003]

Describe the bug
When running the test_mpi_funcs.py, the test will hang infinitely. This does not work on Ubuntu 24.04 running python 3.12.3, but the test passes when running on Ubuntu 20.04 w/ python 3.12.3.

To Reproduce
Steps to reproduce the behavior, for e.g:

1. Clone parsl from repo, on master branch at commit a282d79
2. make virtualenv
3. source .venv/bin/activate
4. make deps
5. pip install -e .
6. pip install radical.pilot
7. Run pytest -v --log-cli-level=DEBUG parsl/tests/test_radical/test_mpi_funcs.py --config local
8. Test does not finish, just hangsExpected behaviorTest doesn't hangEnvironment

• OS: Ubuntu 24.04 LTS
• Python version: 3.12.3
• Parsl version: 1.3.0.dev0

Distributed Environment

• Where are you running the Parsl script from ? Standalone Google Cloud Platform VM
• Where do you need the workers to run ? Local

runinfo_000_ubuntu_2404.tar.gz
runinfo_000_ubuntu_2004.tar.gz

[yadudoc]

Hi @AymenFJA, It looks like the errors reported here are consistently failing in CI. Could you take a look at these failures we are seeing in CI, please?

[AymenFJA]

Hey @yadudoc, @matthewc2003 sorry for the late response. I have 2 questions:

1- Are the Google instances both the same size? i.e. the number of the vcpus/cores
2-Do you see a radical.pilot.sandbox in your /home/mchun078? if so can you also attach the session folders (should be with exact same session id as the one generated in /home/mchun078/parsl/pytest-parsl/parsltest-current/runinfo/000/rpex.session.instance-20250505-230515.c.ucr-ursa-major-hpc-club.internal.mchun078.020214.0012).

I do not see any error in the logs that you attached, besides a KeyboardInterrupt signal. Most likely, the issue is on the agent side (i.e., the folder generated under /home/mchun078/radical.pilot.sandbox)

[matthewc2003]

Hi @AymenFJA, the vCPU count (8) and architecture (AMD Genoa) are identical between the two instances. I have attached the session folder for 0012. I hope this helps!

rpex_session_20250505.tar.gz

[AymenFJA]

@matthewc2003 indeed helpful thanks. Check the following:

746514707.447 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq tasks: 1
1746514707.447 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq task.000000 0 - task pulled
1746514707.451 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq task.000000 1 - task send to 0 [0, 1]
1746514707.452 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq task.000000 1 - task send to 1 [0, 1]
1746514707.453 : rpex.worker.000000   : 15396 : 130104810264256 : ERROR    : recv err  task.000000  to  0
Traceback (most recent call last):
  File "/home/mchun078/parsl/.venv/lib/python3.12/site-packages/radical/pilot/raptor/worker_mpi.py", line 447, in run
    out, err, ret, val, exc = self._dispatch(task)
                              ^^^^^^^^^^^^^^^^^^^^
  File "/home/mchun078/parsl/.venv/lib/python3.12/site-packages/radical/pilot/raptor/worker_mpi.py", line 507, in _dispatch
    return self._dispatch_mpi(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mchun078/parsl/.venv/lib/python3.12/site-packages/radical/pilot/raptor/worker_mpi.py", line 521, in _dispatch_mpi
    group = self._group.Incl(task['ranks'])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/mpi4py/MPI.src/Group.pyx", line 164, in mpi4py.MPI.Group.Incl
mpi4py.MPI.Exception: Invalid rank, error stack:
internal_Group_incl(45482).......: MPI_Group_incl(group=0x88000000, n=2, ranks=0x76549db7b620, newgroup=0x765498f43320) failed
MPIR_Group_check_valid_ranks(240): Invalid rank in rank array at index 1; value is 1 but must be in the range 0 to 0
1746514707.454 : rpex.worker.000000   : 15396 : 130104416003776 : DEBUG    : rrq task.000000 0 [[0, 1]]
1746514707.457 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq tasks: 1
1746514707.457 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq task.000001 0 - task pulled
1746514707.457 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq task.000001 1 - task send to 2 [2, 3, 4]
1746514707.458 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq task.000001 1 - task send to 3 [2, 3, 4]
1746514707.458 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq task.000001 1 - task send to 4 [2, 3, 4]
1746514707.467 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq tasks: 1
1746514707.468 : rpex.worker.000000   : 15396 : 130104424396480 : DEBUG    : wtq task.000002 0 - task pulled
1746514707.478 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq tasks: 1
1746514707.478 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq task.000003 0 - task pulled
1746514707.482 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq task.000003 1 - task send to 0 [0, 1, 2, 3, 4]
1746514707.482 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq task.000003 1 - task send to 1 [0, 1, 2, 3, 4]
1746514707.482 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq task.000003 1 - task send to 2 [0, 1, 2, 3, 4]
1746514707.482 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq task.000003 1 - task send to 3 [0, 1, 2, 3, 4]
1746514707.483 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq task.000003 1 - task send to 4 [0, 1, 2, 3, 4]
1746514707.484 : rpex.worker.000000   : 15389 : 131909166290624 : ERROR    : recv err  task.000003  to  0
Traceback (most recent call last):
  File "/home/mchun078/parsl/.venv/lib/python3.12/site-packages/radical/pilot/raptor/worker_mpi.py", line 447, in run
    out, err, ret, val, exc = self._dispatch(task)
                              ^^^^^^^^^^^^^^^^^^^^
  File "/home/mchun078/parsl/.venv/lib/python3.12/site-packages/radical/pilot/raptor/worker_mpi.py", line 507, in _dispatch
    return self._dispatch_mpi(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mchun078/parsl/.venv/lib/python3.12/site-packages/radical/pilot/raptor/worker_mpi.py", line 521, in _dispatch_mpi
    group = self._group.Incl(task['ranks'])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/mpi4py/MPI.src/Group.pyx", line 164, in mpi4py.MPI.Group.Incl
mpi4py.MPI.Exception: Invalid rank, error stack:
internal_Group_incl(45482).......: MPI_Group_incl(group=0x88000000, n=5, ranks=0x77f8b5008c70, newgroup=0x77f8b500b280) failed
MPIR_Group_check_valid_ranks(240): Invalid rank in rank array at index 1; value is 1 but must be in the range 0 to 0
1746514707.485 : rpex.worker.000000   : 15389 : 131908511987392 : DEBUG    : rrq task.000003 0 [[0, 1, 2, 3, 4]]
1746514707.488 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq tasks: 1
1746514707.488 : rpex.worker.000000   : 15389 : 131908780422848 : DEBUG    : wtq task.000004 0 - task pulled  

The behaviour of running on Ubuntu 20 and not running on Ubuntu 24 confirms that the issue is not from Radical.Pilot. I will discuss this with our team now and will update the ticket soon. In the meantime, can you execute the following commands on both machines if possible and paste the output here, please:

mpiexec --version

mpiexec -n 4 python3 -c "from mpi4py import MPI; print(f'Rank {MPI.COMM_WORLD.Get_rank()} of {MPI.COMM_WORLD.Get_size()}')"

[matthewc2003]

Ubuntu 24.04

mchun078@instance-20250505-230515:~/parsl$ !175
source .venv/bin/activate
(.venv) mchun078@instance-20250505-230515:~/parsl$ mpiexec --version
HYDRA build details:
    Version:                                 4.2.0
    Release Date:                            Fri Feb  9 12:29:21 CST 2024
    CC:                              gcc -Wdate-time -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro 
    Configure options:                       '--with-hwloc-prefix=/usr' '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3' 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Demux engines available:                 poll select
(.venv) mchun078@instance-20250505-230515:~/parsl$ mpiexec -n 4 python3 -c "from mpi4py import MPI; print(f'Rank {MPI.COMM_WORLD.Get_rank()} of {MPI.COMM_WORLD.Get_size()}')"
Rank 0 of 1
Rank 0 of 1
Rank 0 of 1
Rank 0 of 1
(.venv) mchun078@instance-20250505-230515:~/parsl$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
(.venv) mchun078@instance-20250505-230515:~/parsl$ 

Ubuntu 20.04

mchun078@instance-20250505-20250506-064434:~/parsl$ source .venv/bin/activate
(.venv) mchun078@instance-20250505-20250506-064434:~/parsl$ mpiexec --version
HYDRA build details:
    Version:                                 3.3.2
    Release Date:                            Tue Nov 12 21:23:16 CST 2019
    CC:                              gcc   -Wl,-Bsymbolic-functions -Wl,-z,relro 
    CXX:                             g++   -Wl,-Bsymbolic-functions -Wl,-z,relro 
    F77:                             f77  -Wl,-Bsymbolic-functions -Wl,-z,relro 
    F90:                             f95  -Wl,-Bsymbolic-functions -Wl,-z,relro 
    Configure options:                       '--disable-option-checking' '--prefix=/usr' '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-libfabric' '--enable-shared' '--enable-fortran=all' '--disable-rpath' '--disable-wrapper-rpath' '--sysconfdir=/etc/mpich' '--libdir=/usr/lib/x86_64-linux-gnu' '--includedir=/usr/include/x86_64-linux-gnu/mpich' '--docdir=/usr/share/doc/mpich' 'CPPFLAGS= -Wdate-time -D_FORTIFY_SOURCE=2 -I/build/mpich-VeuB8Z/mpich-3.3.2/src/mpl/include -I/build/mpich-VeuB8Z/mpich-3.3.2/src/mpl/include -I/build/mpich-VeuB8Z/mpich-3.3.2/src/openpa/src -I/build/mpich-VeuB8Z/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/build/mpich-VeuB8Z/mpich-3.3.2/src/mpi/romio/include' 'CFLAGS= -g -O2 -fdebug-prefix-map=/build/mpich-VeuB8Z/mpich-3.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -O2' 'CXXFLAGS= -g -O2 -fdebug-prefix-map=/build/mpich-VeuB8Z/mpich-3.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -O2' 'FFLAGS= -g -O2 -fdebug-prefix-map=/build/mpich-VeuB8Z/mpich-3.3.2=. -fstack-protector-strong -O2' 'FCFLAGS= -g -O2 -fdebug-prefix-map=/build/mpich-VeuB8Z/mpich-3.3.2=. -fstack-protector-strong -cpp -O2' 'BASH_SHELL=/bin/bash' 'build_alias=x86_64-linux-gnu' 'MPICHLIB_CFLAGS=-g -O2 -fdebug-prefix-map=/build/mpich-VeuB8Z/mpich-3.3.2=. -fstack-protector-strong -Wformat -Werror=format-security' 'MPICHLIB_CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'MPICHLIB_CXXFLAGS=-g -O2 -fdebug-prefix-map=/build/mpich-VeuB8Z/mpich-3.3.2=. -fstack-protector-strong -Wformat -Werror=format-security' 'MPICHLIB_FFLAGS=-g -O2 -fdebug-prefix-map=/build/mpich-VeuB8Z/mpich-3.3.2=. -fstack-protector-strong' 'MPICHLIB_FCFLAGS=-g -O2 -fdebug-prefix-map=/build/mpich-VeuB8Z/mpich-3.3.2=. -fstack-protector-strong -cpp' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'FC=f95' 'F77=f77' 'MPILIBNAME=mpich' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'LIBS=' 'MPLLIBNAME=mpl'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:       
    Demux engines available:                 poll select
(.venv) mchun078@instance-20250505-20250506-064434:~/parsl$ mpiexec -n 4 python3 -c "from mpi4py import MPI; print(f'Rank {MPI.COMM_WORLD.Get_rank()} of {MPI.COMM_WORLD.Get_size()}')"
Rank 0 of 4
Rank 1 of 4
Rank 2 of 4
Rank 3 of 4
(.venv) mchun078@instance-20250505-20250506-064434:~/parsl$ 

[AymenFJA]

Hey @matthewc2003 @yadudoc, thanks for the output, this is my conclusion:

My conclusion is confirmed. The issue you are facing is not an RP problem, it is the MPI deployment. On Ubuntu 24.04, the MPI world only sees 1 rank

Rank 0 of 1
Rank 0 of 1
Rank 0 of 1
Rank 0 of 1 

And on Ubuntu 20 it sees 4 ranks:

Rank 0 of 4
Rank 1 of 4
Rank 2 of 4
Rank 3 of 4

So the issue is that the MPI setup on 24.04 machine is broken

When the task arrives, RP creates a communicator for each task with N ranks. It could not do that on 24.04 as the world size was 1.

I hope this is clear. LMK if you need additional help.

[yadudoc]

@matthewc2003 It looks like the MPI environment is messed up here. If this is an issue with mpi4py we could try the same approach that we took recently with the WorkQueueExecutor: Split out these tests to a separate CI action that relies on Conda for environment management. Generally the mpi4py builds that come with conda is more reliable.