Modify Omega suites

[cbegeman]

The nightly and pr suites for omega have been updated so they contain the same tests, all of which pass. The one caveat is that sometimes the icos 120km seems to fail for IO reasons but it wasn't consistent enough for me to exclude. The nightly and pr suite names are also updated to include the ocean model name.

[xylar]

@cbegeman, this looks good. I just want to hold off on merging for a tiny bit because I think E3SM-Project/Omega#325 might have fixed the I/O issue finally. That would mean we could add back the highest resolutions here, but still keep the other changes.

[cbegeman]

@cbegeman, this looks good. I just want to hold off on merging for a tiny bit because I think E3SM-Project/Omega#325 might have fixed the I/O issue finally. That would mean we could add back the highest resolutions here, but still keep the other changes.

@xylar Great! I'm glad your testing is panning out so far!

[xylar]

While E3SM-Project/Omega#325 should help, it hasn't solved the problem completely. I'm still seeing it on craygnu-mphipcc on Frontier. So I think we should proceed with this change as it is.

[xylar]

Looks good to me. Thanks @cbegeman!

[xylar]

@cbegeman, I'm seeing order-of-convergence errors in ocean/spherical/icos/rotation_2d :

Order of convergence for tracer1: 1.614
Order of convergence for tracer2: 1.662
Order of convergence for tracer3: 0.275
Error: order of convergence for tracer3
  0.275 < min tolerance 0.3

It seems likely this came from dropping the highest res here. What are your thoughts?

[cbegeman]

@xylar Which machine, compiler did you run that on? I did revise some of the convergence rates downward based on the chrysalis, intel results but we might need to run on all supported machines to figure out the minimum acceptable value. I recall this being the case when I first developed the sphere_transport suite

[xylar]

That was on chrysalis with intel as well.

[cbegeman]

I'll look into it further when chrys's file system is up but this is what I got:

Order of convergence for tracer1: 0.615
Error: order of convergence for tracer1
0.615 < min tolerance 1.0
Order of convergence for tracer2: 1.497
Error: order of convergence for tracer2
1.497 < min tolerance 1.6
Order of convergence for tracer3: 0.342

[xylar]

That's really surprising to me. Did you successfully run the whole omega_pr suite?

[cbegeman]

@xylar I reproduced your icos/rotation_2d convergence rates. I ran qu/rotation_2d to see if that's where I got those convergence rates. I got fails with the 90km qu mesh so ran without it and it's plausible based on tracer2 and tracer3 rates but I don't know what's happening with tracer1. We might need to drop the 90km mesh for now and figure this out. I did intend all tests to pass.

Order of convergence for tracer1: -0.486
Error: order of convergence for tracer1
  -0.486 < min tolerance 0.6
Order of convergence for tracer2: 1.384
Error: order of convergence for tracer2
  1.384 < min tolerance 1.4
Order of convergence for tracer3: 0.352

[xylar]

It is surprising that I am seeing failures on both different tests and different tracers than you are on Chrysalis with Intel. I can't come up with an explanation for why this would be.

I see:

$ cd /lcrc/group/e3sm/ac.xylar/polaris_0.9/chrysalis/test_20260202/omega_update/03_new_329
$ cat case_outputs/ocean_spherical_icos_rotation_2d.log

Order of convergence for tracer1: 1.614
Order of convergence for tracer2: 1.662
Order of convergence for tracer3: 0.275
Error: order of convergence for tracer3
  0.275 < min tolerance 0.3

[cbegeman]

@xylar I reproduced your icos/rotation_2d convergence rates.

FYI