run openmpi programs under flux using PRRTE

[garlick]

Problem: although flux-pmix is probably the main way people will run openmpi jobs in Flux, openmpi comes with its own launcher called PRRTE which could be bootstrapped within a Flux instance. There may be some users who have workflows that require PRRTE.

@hppritcha has made an attempt at adding Flux support to PRRTE and has put up a pull request here:

hppritcha/prrte#97

He would like to get some feedback on the approach.

@hppritcha - would you mind telling us how we can test this in flux? I assume we could pull your branch and run some command line in a Flux instance to demonstrate a hello world mpi application or something?

It might also be helpful to have a couple paragraphs on what your proposed changes actually do and why it is needed (thanks)

[garlick]

See also

• Running PRRTE inside a Flux allocation #3539

[hppritcha]

The contents of that PR only enable prrte to be informed about the flu allocation under which a prte or pretrun command is running, and not have to resort to some hostfile generation script. So the testing I have done for this is merely running commands like hostname and checking that the display allocation information looks reasonable:

prterun --display allocation -n 4 --mapby ppr:2:node hostname

for example.

one can also test with a PMIX client app (for instance an MPI app built using Open MPI) but there are numerous caveats and disclaimers.

For now, prterun is using the ssh process lifecycle (aka launch) manager (PLM), which has some limitations for complicated env. setups.

See https://docs.prrte.org/en/latest/launching-apps/ssh.html

Also the ssh launch mechanism does not propagate any env variable settings (except for those related to PRRTE/PMIX/OMPI) to the shell running on the remote node(s). For example, if there's some vital info stored in LD_LIBRARY_PATH in the shell where prte/prterun is started (like for instance where the "correct" libfabric.so is for Open MPI to use), then that info needs to be handled by the user either by wrapping the app with a script or using .bashrc, etc. settings..

I have been told that at some point in the near future that owing to SS11 credentials, we may need to implement an alternative process launch mechanism (kind of like there is for HPE PALS, SLURM, etc.) but that will be a different PR. If such a PLM needs to be implemented, the above mentioned env. variable issues will hopefully become moot.

[garlick]

Before we go further on this, can you remind me what problem this new feature of PRRTE is solving?

[hppritcha]

Don't have to use explicit hostfiles

[garlick]

I mean how does it benefit flux and/or openmpi users?

[hppritcha]

Open MPI 5 and later need advanced pmix server features like group support for some MPI 4 features etc.

Prrte pmix server implements these advanced pmix server features. Having to use explicit hostfiles on flux managed systems to use prrte would be a major annoyance for users of openmpi 5 or newer.

[garlick]

OK. Regardless of what we do here, we need to get an issue open in flux-pmix on the specific advanced features that aren't working. In theory, we should have or be able to get feature parity with PRRTE over there since both PRRTE and flux-pmix embed the same OpenPMIX server code.

[garlick]

So it seems like it will be required to launch PRRTE as a flux job rather than using ssh. Would the hostfile still be required in that case?

BTW for others, in this context, I think a hostfile is a table of hostnames vs maximum number of "slots" as described here:

https://docs.open-mpi.org/en/main/launching-apps/scheduling.html#scheduling-with-the-hostfile-option

The proposed work derives that information from resource.R, using the core count as the slot count value.

[grondo]

Does resource.R always match the schedulable resources in an instance? If some targets are excluded they appear in resource.R but are not available to the scheduler.

Running PRRTE as a job and using the job's R would have at least a couple benefits:

• excluded and other unavailable resources would not be assigned to the job, so PRRTE would naturally not see them either.
• the PRRTE job's resources would be marked allocated in the instance, so there wouldn't be the potential for other Flux jobs utilizing the resources given to PRRTE.

As noted, this would allow sharing of resources between PRRTE and Flux jobs (though I'm not sure if that's a use case). It might also allow running PRRTE at the top level of a multiuser instance if that's of interest.

[grondo]

FYI - A simple command to generate what I think is a valid prterun hostfile would be something like:

$ flux exec sh -c 'echo $(hostname) nslots=$(nproc)'
fluke129 nslots=2
fluke132 nslots=2
fluke130 nslots=2
fluke131 nslots=2

If that's helpful for now.