Here we provide instructions on how to set up spack to build applications with Intel-MPI on the FASRC Cannon cluster. Intel MPI Library is now included in the Intel oneAPI HPC Toolkit.
The first step involves setting spack to use the Intel compiler, which is provided as a software module. This follows similar procedure to that of adding the GCC compiler.
$ module load intel/23.0.0-fasrc01
$ which icc
/n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/icc$ spack compiler addIf you run the command spack compilers, you will see that the following 3 compilers have been added:
$ spack compilers
...
-- dpcpp rocky8-x86_64 ------------------------------------------
[email protected]
-- intel rocky8-x86_64 ------------------------------------------
[email protected]
-- oneapi rocky8-x86_64 -----------------------------------------
[email protected]Use your favorite text editor, e.g., Vim, Emacs, VSCode, etc., to edit the compiler configuration YAML file ~/.spack/linux/compilers.yaml, e.g.,
$ vi ~/.spack/linux/compilers.yamlEach -compiler: section in this file is similar to the below:
- compiler:
spec: [email protected]
paths:
cc: /n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/icc
cxx: /n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/icpc
f77: /n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/ifort
fc: /n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/ifort
flags: {}
operating_system: rocky8
target: x86_64
modules: []
environment: {}
extra_rpaths: []Note: Here we focus specifically on the
[email protected]compiler as it is required by the Intel MPI Library.
We have to edit the modules: [] line to read
modules: [intel/23.0.0-fasrc01]and save the compilers.yaml file.
We can display the configuration of a specific compiler by the spack compiler info command, e.g.,
$ spack compiler info [email protected]
[email protected]:
paths:
cc = /n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/icc
cxx = /n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/icpc
f77 = /n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/ifort
fc = /n/sw/intel-oneapi-2023/compiler/2023.0.0/linux/bin/intel64/ifort
modules = ['intel/23.0.0-fasrc01']
operating system = rocky8Use your favorite text editor, e.g., Vim, Emacs, VSCode, etc., to edit the packages configuration YAML file ~/.spack/packages.yaml, e.g.,
$ vi ~/.spack/packages.yamlNote: If the file
~/.spack/packages.yamldoes not exist, you will need to create it.
Include the following contents:
packages:
intel-oneapi-mpi:
externals:
- spec: [email protected]%[email protected]
prefix: /n/sw/intel-oneapi-2023
buildable: falseOnce spack is configured to use Intel MPI, it can be used to build packages with it. The below example shows how to install HDF5 version 1.13.2 with [email protected] and [email protected].
You can first test this using the spack spec command to show how the spec is concretized:
$ spack spec [email protected]%[email protected]+mpi+fortran+cxx+hl+threadsafe ^ [email protected]%[email protected]
Input spec
--------------------------------
[email protected]%[email protected]+cxx+fortran+hl+mpi+threadsafe
^[email protected]%[email protected]
Concretized
--------------------------------
[email protected]%[email protected]+cxx+fortran+hl~ipo~java+mpi+shared~szip+threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo arch=linux-rocky8-icelake
^[email protected]%[email protected]~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-rocky8-icelake
^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools arch=linux-rocky8-icelake
^[email protected]%[email protected]~docs~shared build_system=generic certs=mozilla arch=linux-rocky8-icelake
^ca-certificates-mozilla@2022-10-11%[email protected] build_system=generic arch=linux-rocky8-icelake
^[email protected]%[email protected]+cpanm+shared+threads build_system=generic arch=linux-rocky8-icelake
^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky8-icelake
^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=linux-rocky8-icelake
^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-icelake
^[email protected]%[email protected] build_system=autotools libs=shared,static arch=linux-rocky8-icelake
^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-icelake
^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-icelake
^[email protected]%[email protected] cflags="-gcc-name=/usr/bin/gcc" ~external-libfabric~generic-names~ilp64 build_system=generic arch=linux-rocky8-icelake
^[email protected]%[email protected] build_system=autotools arch=linux-rocky8-icelake
^[email protected]%[email protected]+optimize+pic+shared build_system=makefile arch=linux-rocky8-icelakNext, you can build it:
$ spack install [email protected]%[email protected]+mpi+fortran+cxx+hl+threadsafe ^ [email protected]%[email protected]
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/pkgconf-1.8.0-p4zr5nbn2qfnm2ezj3yuna7z7ldxugfo
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/ca-certificates-mozilla-2022-10-11-rlli5fmtpjaatq4lseetdudmhhcilc43
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/berkeley-db-18.1.40-m6r6onkubysvl3gxtw7t7tpghoq6x6g6
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/libiconv-1.16-iidmiswunjmyfgqv5im7dbq3vxmfugvn
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/zlib-1.2.13-admgboe6z4bq56lnw3owqrwqydoh2qkv
[+] /n/sw/intel-oneapi-2023 (external intel-oneapi-mpi-2021.8.0-xfvjrn2fuyum7xtnyz7g3gjv3c5d27tf)
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/ncurses-6.3-wq3qyyg5nbr3r4rbpxtm2w6sz2asqwmf
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/diffutils-3.8-idtn6tigrq7qhdqlofjawd5jax6ugen6
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/readline-8.1.2-lu2zjeb2y7y7k6trks5bkgswzc5mhz6h
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/bzip2-1.0.8-q4kqkohouj2iwivc2kav2imqmkkime3k
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/gdbm-1.23-h5r23zyhy5plttfjopj6hfn5nxg3526q
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/perl-5.36.0-moqx3wcrd3zgpgm4q27phfuxoymdhnkg
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/openssl-1.1.1s-rjscbgskagds5vtgqfis7ynma42o2zn5
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/cmake-3.24.3-zj2opbmjgzlp6x3ww5t2m2s6a5fko6sh
==> Installing hdf5-1.13.2-2yyrm7ssqifdsgwjthxbmp3az3pitkxj
==> No binary for hdf5-1.13.2-2yyrm7ssqifdsgwjthxbmp3az3pitkxj found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/01/01643fa5b37dba7be7c4db6bbf3c5d07adf5c1fa17dbfaaa632a279b1b2f06da.tar.gz
==> Ran patch() for hdf5
==> hdf5: Executing phase: 'cmake'
==> hdf5: Executing phase: 'build'
==> hdf5: Executing phase: 'install'
==> hdf5: Successfully installed hdf5-1.13.2-2yyrm7ssqifdsgwjthxbmp3az3pitkxj
Fetch: 1.49s. Build: 1m 58.82s. Total: 2m 0.32s.
[+] /builds/pkrastev/Spack/spack/opt/spack/linux-rocky8-icelake/intel-2021.8.0/hdf5-1.13.2-2yyrm7ssqifdsgwjthxbmp3az3pitkxj