sptrsv cholmod and superlu benchmarks

These benchmarks are located in /path/to/kokkos-kernels/perf_test/sparse/KokkosSparse_sptrsv_{superlu,cholmod}.cpp. The following tpls must be enabled to build these benchmarks: cholmod,superlu,cblas,blas,lapacke. If you're targeting a CUDA device, you must also enabled the cusparse tpl.

Building

Know which architecture you're targeting before compiling. Below we set KOKKOS_ARCH to target Power8 CPUs and Pascal60 GPUs.

Setup your environment

cd /path/to/my/kokkos-kernels/build/directory

### kokkos and kokkos-kernels source and install paths ###
KOKKOS_BASE=/path/to/Kokkos
KOKKOSKERNELS_PATH=$KOKKOS_BASE/kokkos-kernels #path to kokkos source
KOKKOSKERNELS_INSTALL=$KOKKOSKERNELS_PATH/install #path to where kokkos-kernels should be installed

KOKKOSKERNELS_LAYOUTS=LayoutLeft #the layout types to instantiate.

KOKKOSKERNELS_SCALARS='double' #the scalar type to instantiate.
#KOKKOSKERNELS_SCALARS='"complex<double>"'

KOKKOSKERNELS_ORDINALS=int #ordinal types to instantiate
KOKKOSKERNELS_OFFSETS=int #offset types to instantiate

KOKKOSKERNELS_OPTIONS=eti-only #options for kokkoskernels  

#### Point the build system to the TPLs ####
CHOLDMODDIR=$SUITESPARSE_ROOT # This must be >= v5.1.1
SUPERLUDIR=$SUPERLU_ROOT # I used v5.1.1
METIS_PATH=$METIS_ROOT # I used v5.0.1

CUDA_PATH=$CUDA_ROOT # v10.1.105

EXTRA_CXX_FLAGS="-I${METIS_PATH}/include"

# Optionally enable profiling
ENABLE_OPTIONS="-DKOKKOS_SPTRSV_SUPERNODE_PROFILE"

# CPU
#KOKKOS_DEVICES=OpenMP
#KOKKOS_ARCH=Power8
#CXX=g++
#CXXFLAGS="-Wall -pedantic -Werror -O3 -fopenmp -std=c++11 -g -Wshadow -Wsign-compare -Wtype-limits -Wuninitialized $EXTRA_CXX_FLAGS $ENABLE_OPTIONS"
#EXTRA_OPTIONS="--with-tpls=cholmod,superlu,cblas,blas,lapacke"

# GPU
KOKKOS_DEVICES='"Cuda,OpenMP"'
KOKKOS_ARCH='"Power8,Pascal60"'
WITH_CUDA="--with-cuda=${CUDA_ROOT}"
CXX=${KOKKOS_PATH}/bin/nvcc_wrapper
KOKKOS_CUDA_OPTIONS="enable_lambda"
WITH_CUDA_OPTIONS="--with-cuda-options=${KOKKOS_CUDA_OPTIONS}"
CXXFLAGS="-Wall -pedantic -Werror -O3 -fopenmp -std=c++11 -g -Wshadow -Wsign-compare -Wtype-limits -Wuninitialized $EXTRA_CXX_FLAGS $ENABLE_OPTIONS"
EXTRA_OPTIONS="--with-tpls=cholmod,superlu,cblas,blas,lapacke,cusparse"

Configure

source $KOKKOSKERNELS_PATH/scripts/generate_makefile.bash --kokkoskernels-path=${KOKKOSKERNELS_PATH} --with-scalars=${KOKKOSKERNELS_SCALARS} --with-ordinals=${KOKKOSKERNELS_ORDINALS} --with-offsets=${KOKKOSKERNELS_OFFSETS} \
    --kokkos-path=${KOKKOS_PATH} --with-devices=${KOKKOS_DEVICES} $WITH_CUDA --arch=${KOKKOS_ARCH} --compiler=${CXX} --with-options=${KOKKOSKERNELS_OPTIONS}  --cxxflags="${CXXFLAGS}" \
    $WITH_CUDA_OPTIONS --prefix=${KOKKOSKERNELS_INSTALL} ${EXTRA_OPTIONS}

Compile kokkos-kernels

make install-lib -j16 &> make.out

Compile the perf tests

cd perf_test
make -j16 &> make.out

Running

You will also need some matrix market input files. These can be found here. For cholmod, you will need to filter files for "Special Structure->Symmetric" and "Positive Definite->Yes".

Ensure you're on a compute node with Power8 CPUs and Pascal60 GPUs, then see KokkosSparse_sptrsv_superlu.exe --help and KokkosSparse_sptrsv_cholmod.exe --help.