Build and Run Recipes

All recipes assume the code was downloaded recursively and you are in a clean directory one level into the build. E.g.,

git clone --recursive [email protected]:lanl/phoebus.git
mkdir -p phoebus/build
cd phoebus/build

Unit tests

cmake -DPHOEBUS_ENABLE_UNIT_TESTS=ON ..
make -j
make test

Blast wave 3D

Many small meshblocks. Lots of mesh refinement.

cmake ..
make -j
./src/phoebus -i ../inputs/blast_wave_3d

Linear modes 3D

Uniform grid problem. Not very stressful on the infrastructure.

cmake ..
make -j
./src/phoebus -i ../inputs/blast_wave_3d

Torus

Realistic production problem

cmake -DPHOEBUS_GEOMETRY=FMKS -DPHOEBUS_CACHE_GEOMETRY=ON .. 
make -j
./src/phoebus -i ../inputs/torus.pin

Disabling HDF5

If outputs are not desired (which is possible) compile with the additional cmake option

-DPHOEBUS_ENABLE_HDF5=OFF

Disabling MPI

If MPI is not desired, configure with -DPHOEBUS_ENABLE_HDF5=OFF

Enable OpenMP

To run with OpenMP looping, use -DPHOEBUS_ENABLE_OPENMP=ON

Disabling HDF5 Compression

You may need to configure with -DPARTHENON_DISABLE_HDF5_COMPRESSION=ON in some circumstances. The cmake error message will tell you.

MPI

Phoebus is inherently MPI-capable. If you built the code with MPI (the default) you can simply prefix the run command with mpirun -n N where N is the number of ranks.

Cuda

You can enable cuda offloading with -DPHOEBUS_ENABLE_CUDA=ON\

Changing compilers

On an HPC system, it should be enough to change the modules loaded. But if it isn't, use

-DCMAKE_CXX_COMPILER=<path/to/compiler>

You can also tell kokkos to optimize for a particular architecture with -DKokkos_ENABLE_<architecture_name>=ON.

Changing runtime parameters

You can change input parameters on the command line, rather than the input deck. For example, tro run the blast wave 3D for only 10 cycles, since the simulation is likely expensive, use the following run command:

./src/phoebus -i ../inputs/blast_wave_3d parthenon/time/nlim=10

where parthenon/time/nlim is the maximum number of cycles before the simulation ends.

Some other parameters that may be worth modifying are:

• parthenon/mesh/nx1, parthenon/mesh/nx2, and parthenon/mesh/nx3. Integers. These control the total problem size and specify the resolution of the base grid
• parthenon/meshblock/nx1, parthenon/meshblock/nx2, and parthenon/meshblock/nx3. Integers. These control the size of meshblocks, which are the pieces out of which the AMR mesh is constructed. parthenon/mesh/nx* must be evenly divisible by partheon/meshblock/nx*.
• parthenon/mesh/pack_size. Integer. Controls how meshblocks are coalesced into loops. -1 is the default and means all meshblocks on an MPI rank are coalesced. 1 means loops are per meshblock. N means loops contain N meshblocks.