Building Albany on Ride or White

These build instructions are for compiling Albany on the Ride or White IBM Power8 GPU clusters at Sandia National Laboratories. They have only been tested for the Aeras and FELIX applications but can be adapted for other Albany applications.

Download Trilinos and Albany to the home directory by using git:

cd ${HOME}
module load git
git clone https://github.com/trilinos/Trilinos.git
git clone https://github.com/SNLComputation/Albany.git

Building Albany with Kokkos::Serial

The following build instructions are for building Albany with Kokkos::Serial.

Environment setup

Load the following modules:

module purge
module load git cmake
module load openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load netcdf-f/4.4.4/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load pnetcdf/1.6.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load openblas/0.2.19/gcc/5.4.0

It's best to create a shell script to run these commands before building. An example script can be found under ${HOME}/Albany/doc/ride-white/modules.sh. Execute the script by using source:

source ${HOME}/Albany/doc/ride-white/modules.sh

Building Trilinos

Make a build directory for Trilinos and copy over the configuration file provided in ${HOME}/Albany/doc/ride-white/do-cmake-trilinos-serial:

cd ${HOME}/Trilinos/
mkdir build-kokkos-serial
cd build-kokkos-serial/
cp ${HOME}/Albany/doc/do-cmake-trilinos-serial .

(Optional) Before building Trilinos, log into a compute node by using the LSF command bsub -Is bash.

Build and install Trilinos by using the configuration file provided:

source do-cmake-trilinos-serial
make -j
make install

(Optional) Use make -j 2>&1 | tee output.txt to output the results of make into a file.

Building Albany

Make a build directory for Albany and copy over the configuration file provided in ${HOME}/Albany/doc/ride-white/do-cmake-albany:

cd ${HOME}/Albany/
mkdir build-kokkos-serial
cd build-kokkos-serial/
cp ${HOME}/Albany/doc/ride-white/do-cmake-albany .

Make sure to modify the file to point to your Trilinos build directory and build:

source do-cmake-albany
make -j

(Optional) Once building is complete, exit the compute node with the command exit.

Building Albany with Kokkos::OpenMP

The same build instructions for Albany with Kokkos::Serial can be used to build Albany with Kokkos::OpenMP except that the configuration file ${HOME}/Albany/doc/ride-white/do-cmake-trilinos-openmp needs to be used instead of ${HOME}/Albany/doc/ride-white/do-cmake-trilinos-serial. Also, hwloc needs to be installed locally.

Building hwloc

The hwloc module on the cluster requires cuda to link properly to Trilinos. For an openmp build without cuda, a local copy of hwloc must be installed.

First, create an install directory in the home directory and download and unpack hwloc:

cd ${HOME}
mkdir Install
cd Install/
wget --no-check-certificate https://www.open-mpi.org/software/hwloc/v1.11/downloads/hwloc-1.11.3.tar.gz
tar -xvf hwloc-1.11.3.tar.gz

Before configuring hwloc, make sure to load all appropriate modules by following the enviroment setup section. Then install hwloc locally:

cd ${HOME}/Install/hwloc-1.11.3
./configure --prefix=${HOME}/Install/hwloc-1.11.3 CC=mpicc CXX=mpicxx
make -j
make install

Building Albany with Kokkos::Cuda

The following build instructions are for building Albany with Kokkos::Cuda. The instructions are slightly different depending on the type of GPU you wish to run on. For K40m GPUs, use compute capability 3.5. For K80 GPUs, use compute capability 3.7. For P100 GPUs, use compute capability 6.0.

Environment setup

First, modify ${HOME}/Trilinos/packages/kokkos/config/nvcc_wrapper to use the appropriate compute capability of the target GPU (i.e. Change default_arch="sm_35" to default_arch="sm_37" for K80 GPUs). You may want to create multiple copies of this file for each type of GPU if you're building for multiple GPU types.

Load the following modules:

module purge
module load cmake git
module load openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load netcdf-f/4.4.4/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load pnetcdf/1.6.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load openblas/0.2.19/gcc/5.4.0
module load hwloc/1.11.3/cuda/8.0.44

Set the following compiler and GPU settings. Make sure to modify the OMPI_CXX path to point to the appropriate nvcc_wrapper file:

export OMPI_CXX=${HOME}/Trilinos/packages/kokkos/config/nvcc_wrapper
export CUDA_MANAGED_FORCE_DEVICE_ALLOC=1
export CUDA_LAUNCH_BLOCKING=1

It's best to create a shell script to run these commands before building. An example script can be found under ${HOME}/Albany/doc/ride-white/modules_cuda.sh. Execute the script by using source:

source ${HOME}/Albany/doc/ride-white/modules_cuda.sh

Building Trilinos

Make a build directory for Trilinos and copy over the configuration file provided in ${HOME}/Albany/doc/ride-white/do-cmake-trilinos-cuda:

cd ${HOME}/Trilinos/
mkdir build-kokkos-cuda
cd build-kokkos-cuda/
cp ${HOME}/Albany/doc/do-cmake-trilinos-cuda .

Make sure to modify the configuration file to use the appropriate compute capability of the target GPU (i.e. Change -arch=sm_35 to -arch=sm_37 for K80 GPUs).

(Optional) Before building Trilinos, log into a compute node by using the LSF command bsub -Is bash.

Build and install Trilinos by using the configuration file provided:

source do-cmake-trilinos-cuda
make -j
make install

(Optional) Use make -j 2>&1 | tee output.txt to output the results of make into a file.

Building Albany

Make a build directory for Albany and copy over the configuration file provided in `${HOME}/Albany/doc/ride-white/do-cmake-albany:

cd ${HOME}/Albany/
mkdir build-kokkos-cuda
cd build-kokkos-cuda/
cp ${HOME}/Albany/doc/ride-white/do-cmake-albany .

Make sure to modify the file to point to your Trilinos build directory and build:

source do-cmake-albany
make -j

(Optional) Once building is complete, exit the compute node with the command exit.