Higher order horizontal advection copied from mpas-ocean to Omega.

[overfelt]

Checklist

• Documentation:
  • Design document has been generated and added to the docs
  • User's Guide has been updated
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected
• Building
  • CMake build does not produce any new warnings from changes in this PR
• Testing
  • A comment in the PR documents testing used to verify the changes including any tests that are added/modified/impacted.
  • CTest unit tests for new features have been added per the approved design.
  • Polaris tests for new features have been added per the approved design (and included in a test suite)
  • Unit tests have passed. Please provide a relevant CDash build entry for verification.
  • Polaris test suite has passed
  • Performance related PRs: Please include a relevant PACE experiment link documenting performance before and after.
• Stealth Features
  • If any stealth features are included in the PR, please confirm that they have been documented.

[overfelt]

Testing:
Unit tests for the new functions added to HorzOperators class.
(1) testsecondderivativeoncellDeterminePlanerPatchGeometry - Second derivative on a patch of elements test. This test creates a little patch of elements and defines the geometry from scratch by just hard coding a bunch of coordinates. Then the second derivative calculated by the HorzOperators class is compared to an analytic solution.
(2) testsecondderivativeoncellLeastSquaresFit - The LeastSquaresFit function is tested against an analytic solution.
(3) testsecondderivativeoncellDetermineSphericalPatchGeometry - Is like the testsecondderivativeoncellDeterminePlanerPatchGeometry unit test but on a spherical mesh.
(4) testsecondderivativeoncellconstructor - A trivial unit test that just checks that an instance of the class SecondDerivativeOnCell can be constructed.

[philipwjones]

@overfelt This is failing both the Horz Operators unit tests on Chrysalis and on Frontier (gpu only - cpu passes). The errors are memory errors - invalid pointer in free() on Chrysalis and inaccessible memory space (for array named XP) on Frontier gpu. So something isn't getting cleaned up correctly? I'll start reading through code to review...

[overfelt]

@philipwjones ,I only ran perlmutter gpu. I'll try setting up and running on Chrysalis gpu. Thanks.

[mark-petersen]

@overfelt, the tests also fail on perlmutter, both CPU and GPU, using the gnu compiler.

The following tests FAILED:
	 11 - HORZOPERATORS_PLANE_TEST (Failed)
	 12 - HORZOPERATORS_SPHERE_TEST (Failed)

Here are my instructions. Let me know if you can reproduce these errors. You will need to change to your own paths.

perlmutter CPU:

######### perlmutter CPU
#export CODEDIR=omega-develop
export CODEDIR=opr
export RUNDIR=test_omega_cpu

cd /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}
git submodule update --init --recursive components/omega/external/GSW-C components/omega/external/yaml-cpp externals/ekat externals/scorpio cime externals/cpptrace
cd components/omega/

module load cmake
mkdir ${PSCRATCH}/runs/$RUNDIR
cd ${PSCRATCH}/runs/$RUNDIR

rm -rf build
mkdir build
cd build

# compiler options are:
export compiler=gnu
#export compiler=nvidia # not working 250421

export PARMETIS_ROOT=/global/cfs/cdirs/e3sm/software/polaris/pm-cpu/spack/dev_polaris_0_9_0_${compiler}_mpich/var/spack/environments/dev_polaris_0_9_0_${compiler}_mpich/.spack-env/view

# nvidia or gnu compiler:
cmake \
   -DOMEGA_CIME_COMPILER=${compiler} \
   -DOMEGA_BUILD_TYPE=Release \
   -DOMEGA_CIME_MACHINE=pm-cpu \
   -DOMEGA_PARMETIS_ROOT=${PARMETIS_ROOT}\
   -DOMEGA_BUILD_TEST=ON \
   -DOMEGA_VECTOR_LENGTH=1 \
   -Wno-dev \
   -S /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}/components/omega -B .
# note OMEGA_VECTOR_LENGTH=8 fails MPI tests on CPUs.
cd ${PSCRATCH}/runs/$RUNDIR/build
./omega_build.sh

# linking:
cd test
ln -isf /global/homes/m/mpeterse/meshes/omega/O*nc .
cp /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}/components/omega/configs/Default.yml omega.yml

# run test:
salloc --nodes 1 --qos interactive --time 01:00:00 --constraint cpu --account=m4572 # or e3sm
cd ${PSCRATCH}/runs/${RUNDIR}/build

./omega_ctest.sh

perlmutter GPU:

######### perlmutter GPU
salloc --nodes 4 --qos interactive --time 01:00:00 --constraint gpu --tasks-per-node=2 --gpus-per-task 1 --account=m4572_g # or e3sm_g

# perlmutter has nodes with either 40 or 80 gb of high bandwidth memory, and the system defaults to 40. You can ask for 80 gb nodes with the sbatch flag --constraint="gpu&hbm80gb"

export CODEDIR=opr
#export CODEDIR=omega-develop
export RUNDIR=test_omega_gpu
mkdir ${PSCRATCH}/runs/$RUNDIR
cd !$

rm -rf build
mkdir build
cd build
module load cmake

# compiler options are:
export compiler=gnugpu
#export compiler=nvidiagpu

export PARMETIS_ROOT=/global/cfs/cdirs/e3sm/software/polaris/pm-gpu/spack/dev_polaris_0_9_0_${compiler}_mpich/var/spack/environments/dev_polaris_0_9_0_${compiler}_mpich/.spack-env/view
cmake \
   -DOMEGA_CIME_COMPILER=${compiler} \
   -DOMEGA_BUILD_TYPE=Release \
   -DOMEGA_CIME_MACHINE=pm-gpu \
   -DOMEGA_PARMETIS_ROOT=${PARMETIS_ROOT}\
   -DOMEGA_BUILD_TEST=ON \
   -Wno-dev \
   -DOMEGA_MPI_ON_DEVICE:BOOL=OFF \
   -S /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}/components/omega -B .
# needed for compiler bug: OMEGA_MPI_ON_DEVICE:BOOL=OFF. See https://github.com/E3SM-Project/Omega/issues/214
./omega_build.sh

# linking:
cd test
ln -isf /global/homes/m/mpeterse/meshes/omega/O*nc .
cp /global/homes/m/mpeterse/repos/E3SM/${CODEDIR}/components/omega/configs/Default.yml omega.yml

cd ..
./omega_ctest.sh

[philipwjones]

With the recent commit, this now passes CTests on Chrysalis and Frontier (cpu/gpu). Thanks @overfelt

[mark-petersen]

This passes CPU and GPU tests on perlmutter. Due to the merge of #314, this now has some conflicts to be resolved. Please rebase on the current head, and we will proceed with the review after the break. Thanks!

[philipwjones]

I have not done an exhaustive comparison to the paper or MPAS to verify the terms, but I did browse the code somewhat and this appears to be working fine based on convergence testing and passes all unit tests. Some of the code (esp HorzUtil) does not match Omega coding style, but I won't hold this up. At some point in the near future, I think we need to break up the monolithic tendency file.

[brian-oneill]

I ran the ctests successfully on pm-cpu, pm-gpu, and Chrysalis.

I agree with Phil, I think this is fine to move forward. Just FYI on Omega coding style, we're using PascalCase for variable names and camelCase for function names. This PR is just the standard horizontal advection algorithm so it's fine as is, but I think the monotonic advection algortihm is complex enough to warrant moving horizontal advection to it's own file, possibly as a separate module.

[cbegeman]

@overfelt Glad to see this coming along! Let me know when this is ready to run Polaris tests and if you need any help getting those going

[overfelt]

What Polaris tests should I be running? I have only run the spherical cosine_bell test in my development.

[cbegeman]

What Polaris tests should I be running? I have only run the spherical cosine_bell test in my development.

It would be
polaris suite -c ocean -t sphere_transport_with_viz ...

That includes the icos meshes. After those pass, then you can move on to the qu meshes (just make substitutions in the test case name on setup).

These convergence tests include some higher-resolutions meshes that have been hitting IO errors. If you want to bypass those, use the latest omega_pr suite modified just recently here E3SM-Project/polaris#455.

The order of the advection scheme tested will be whatever the default option is in Omega or you'll have to manually change it (until we update Polaris). If that's too much of a pain, let me know and I can automate it in a polaris branch.

And then you can just visually examine the convergence plot and the state visualizations since the analysis PASS/FAIL will be based off of hard-coded orders of convergence (https://github.com/E3SM-Project/polaris/blob/bd87bf281986b9a28ad63370d0ad495140f4d84d/polaris/tasks/ocean/sphere_transport/analysis.py#L76-L79)