Build

Due to the large number of options required to manage building AthenaK and Kokkos on different architectures, the code no longer uses a configure script (as in Athena++), but instead adopts cmake (version 3.0 or later) to manage builds. In-source builds are not allowed; you must create a new build directory in which cmake can be run, for example:

$ mkdir build
$ cd build
$ cmake [options] ../

Before running cmake it is important to ensure the correct compilers, device toolkits (if required), and MPI libraries (if required) are properly loaded (see Requirements).

Kokkos provides many options for controlling cmake, see their documentation. The following are useful cmake options frequently used with AthenaK

Building for different architectures

By default, cmake will build an x86 compatible executable. Thus, if the executable will be run on an x86 compatable processer, no additional options are required:

$ cmake ../

However, often better performance can be achieved by specifying the specific target processor using the -D Kokkos_ARCH_XXX=On option, where XXX specifies the target architecture (see the Architecture Keywords listed on the Kokkos cmake options page. Some examples are given below:

Build for Intel Skylake processor using icpc compiler:

$ cmake -DCMAKE_CXX_COMPILER=icpc -D Kokkos_ARCH_SKX=On ../

Build for for NVidia V100 gpu

$  cmake3 -DKokkos_ENABLE_CUDA=On -DKokkos_ARCH_VOLTA70=On -DCMAKE_CXX_COMPILER=${path_to_code}/kokkos/bin/nvcc_wrapper ../

Note that additional options are required to specify a CUDA backend and nvcc compiler

Build for NVidia A100 gpu

$  cmake3 -DKokkos_ENABLE_CUDA=On -DKokkos_ARCH_AMPERE80=On -DCMAKE_CXX_COMPILER=${path_to_code}/kokkos/bin/nvcc_wrapper ../

To build with custom problem generators

To include user-defined problem generators (located in a /src/pgen/name.cpp file) in the build, use the -D PROBLEM=name option.

To build in debug mode

Include the -D CMAKE_BUILD_TYPE=Debug option

To build with MPI

Use the -D Athena_ENABLE_MPI=ON option.

To build with specific compiler options

Additional compiler options can be included using -D CMAKE_CXX_FLAGS="options". For example

$ cmake -DCMAKE_CXX_COMPILER=icpc -D Kokkos_ARCH_BDW=On -D CMAKE_CXX_FLAGS="-O3 -inline-forceinline -qopenmp-simd -qopt-prefetch=4 -diag-disable 3180 " -D CMAKE_C_FLAGS="-O3 -finline-functions" ../