Add vertical advection

[brian-oneill]

Add the VertAdv module to Omega, providing data structures and methods for vertical advection. This module includes:

• Arrays for storing vertical velocity and tracer fluxes at layer interfaces.
• A method to compute the vertical velocity from the divergence of the horizontal velocity.
• Methods for computing the tendencies of thickness, horizontal velocity, and tracers due to vertical advection. The tracer tendency methods include multiple options for order of accuracy of the flux calculation, as well as algorithmic options for standard advection or flux-corrected transport (FCT).

Additional changes in this PR:

• Extend parallelForOuter with an optional argument to enable the use of scratch arrays in hierarchical parallelism.
• Add the auxiliary variable ProvThickness, which represents thickness after horizontal thickness flux. Further work required to fully implement provisional thickness.
• Remove BottomDepth from HorzMesh, this field was previously added to VertCoord.

This PR builds and passes the unit tests successfully on Chrysalis, pm-cpu, pm-gpu, and Frontier (CPU & GPU).

Checklist

• Documentation:
  • Design document has been generated and added to the docs
  • User's Guide has been updated
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected
• Linting
  • Pre-commit run locally
• Building
  • CMake build does not produce any new warnings from changes in this PR
• Testing
  • Add a comment to the PR titled Testing with the following:
    • Which machines CTest unit tests
      have been run on and indicate that are all passing.
    • The Polaris omega_pr test suite
      has passed, using the Polaris e3sm_submodules/Omega baseline
    • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
    • Indicate "All tests passed" or document failing tests
    • Document testing used to verify the changes including any tests that are added/modified/impacted.
  • New tests:
    • CTest unit tests for new features have been added per the approved design.
    • Polaris tests for new features have been added per the approved design (and included in a test suite)

[mwarusz]

When running VERTADV_TEST with bounds checking enabled I get out-of-bounds errors. The reason is VertCoord::init(false) gets NVertLayers from OmegaMesh.nc and this value is used to create the member arrays of VertAdv in VertAdv::init(). The value of NVertLayers in VertAdv and MaxLayerCell get overwritten later in the convergence test, but the VertAdv member arrays (e.g. VertFlux) don't get resized.

[cbegeman]

Suggested change

	VerticalTracerFluxLimiterEnabled: true
	VerticalTracerFluxLimiterEnable: true

I think this is more consistent with other config options

[cbegeman]

Do we want to use

Omega/components/omega/src/ocn/HorzOperators.h

Line 139 in f2e951a

InterpCellToEdge(const HorzMesh *Mesh);

so that we can easily switch from anisotropic to isotropic interpolation for all variables (here and elsewhere)?

[hyungyukang]

I think this may cause a circular include issue in the headers when initializing with AuxiliaryState. I’d suggest removing this line unless it introduces any other issues.

Suggested change

#include "AuxiliaryState.h"

[hyungyukang]

Suggested change

	// Return if vertical advection tracer tendency not enabled
	if (!TracerVertAdvEnabled)
	return;