Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCN to GLC thermal forcing coupling #6632

Open
wants to merge 22 commits into
base: master
Choose a base branch
from

Conversation

matthewhoffman
Copy link
Contributor

@matthewhoffman matthewhoffman commented Sep 20, 2024

This pull request introduces coupling from OCN to GLC to pass thermal forcing at a prescribed ocean depth (300 m) from MPAS-Ocean to MALI, where MALI uses it to calculate grounded marine melting through an existing 'facemelting' parameterization. Facemelting as a function of ocean thermal state is a critical OCN/GLC coupling for Greenland, but it is relevant to Antarctica as well. This OCN-GLC thermal forcing coupling is activated whenever OCN is present and GLC is active.

This PR has pieces in MPAS-Ocean, the coupler, and MALI:

  • In MPAS-Ocean, a new avgThermalForcingAtCritDepth field is added and calculated. The depth at which to calculate the thermal forcing is namelist configurable. The default is 300 m.
  • In MALI, the field passed from the coupler is collected in ismip6_2dThermalForcing and used in the existing facemelting parameterization. Namelist and streams are updated to use facemelting in all simulations and output relevant variables. For compsets where TF is not being coupled, it will have values of 0 and result in no facemelting being applied; it is safe to have this option generally turned on.
  • In the coupler, a new remapping object is created that handles the TF mapping independently of any other fields passed from OCN to GLC. New mapping files are added. Any coupling variables related to the existing ice-shelf coupling now include the 'shelf' suffix and all coupling variables related to this new thermal-forcing coupling have the 'tf' suffix.
  • A new compset is created that is a standard G-case but with active MALI. New mapping files are added between the relevant grids for the new mapper.
  • The new OCN2GLC_TF_SMAPNAME mapping should be a nearest source to destination mapping with the MPAS-Ocean source mesh masked to only include grid cells that are deeper than the critical depth to avoid extrapolating undefined values. A tool to generate that map file is here: Add script for adding map of sufficiently deep ocean bathymetry to a scrip file MPAS-Dev/MPAS-Tools#578
  • Two new model_grids are added that are compatible with this feature and include the special mapping file required: TL319_IcoswISC30E3r5_gis1to10kmR2 and TL319_oQU240wLI_gis20
  • The new coupling is controlled by a new namelist option in MPAS-Ocean: config_glc_thermal_forcing_coupling_mode. It defaults to 'off' and there are no compsets that explicitly set it for now, making this a stealth feature.

This PR was initially reviewed at E3SM-Ocean-Discussion#94

[NML] for new mappings, new ocean variable
[non-BFB] for configurations with active MALI and X-cases (due to new coupler field)

matthewhoffman and others added 17 commits September 20, 2024 13:24
This commits adds a new field to MPAS-Ocean called
avgThermalForcing300m.  It is calculated as the thermal forcing
(difference between ocean temperature local freezing temperature) at 300
m depth.  It uses the temperature of the layer shallower than 300 m.
This could be replaced with vertical interpolation to get the value
exactly at 300 m.  The TF at 300 m is time averaged over the ocean coupling
interval.
This commit adds the So_tf300 coupler field for passing thermal forcing
at 300m through the coupler.  It also passes the avgThermalForcing300m
added to MPAS-Ocean in the previous commit to this coupler field, and
send it to the ismip6_2dThermalForcing field in MALI as the destination.

Note that a new list of coupler fields called x2g_tf_states_from_ocn
has been added to seq_flds_mod to differentiate this coupling field from
the iceshelf OCN/GLC coupling, which is handled differently.
This commit enables facemelting in MALI so that the new TF field will be
used to calculate a melt rate.  This is safe to enable in general,
because for situations where thermal forcing is not calculated  or not
applicable, it will be zero and facemelting will in turn be zero.  This
commit also updates the MALI output stream to write out the TF and
facemelting fields.

Note that this commit also changes the MALI output stream interval to
daily.  While that might not be the ideal long-term production solution,
it is likely the desired frequency for model development for the
forseeable future.
This commits defines the mapping for thermal forcing from ocean to glc.
It defines a special mapper for thermal forcing state (ocn2glc_tf_smap)
in CIME and MCT xml files.  It also declares and allocates mapper_So2g_tf
in prep_glc_mod but does not initialize or use it yet.

The mapping file requires some special treatment:
* it needs to use nearest neighbor mapping (which differs from most
  state remapping)
* it needs to include grid_imask in the ocean scrip file to only consider
  ocean cells valid if they are deeper than 300m
1. Differnetiate new ocn2glc TF coupling from existing ocn2glcshelf coupling
Create ocn_c2_glc logical in cime_comp_mod.  This controls the new
ocn2glc TF (thermal forcing) coupling separately from the existing
ocn2glcshelf coupling, which has its own ocn_c2_glcshelf flag already.
The new TF coupling will be active whenever ocn is present and glc is
prognostic.  These flags are used to initialize different mappers
independent of each other in prep_glc_init.  Those flags are also now
passed to prep_glc_calc_o2x_gx to control which mapping actually occurs.

2. Implement prep_glc_mrg_ocn
The existing ocn2glcshelf coupling handled a number of operations in
unusual ways because the fluxes themselves are calculated in the
coupler, so it was missing some of the standard operations for simply
passing fields between components.  As such, adding the new TF coupling
requires creating a prep_glc_mrg_ocn routine, which is responsible for
transferring fields from o2x_g to x2g_g arrays.  This routine was copied
closely from the existing prep_glc_mrg_lnd routine.

With this commit, the ocean TF field is successfully passed to MALI.
The entirety of existing ocn->glc coupling was for the ice-shelf
coupling.  To better differentiate the coupling in this branch based on
thermal forcing, this commit ensures there is either a 'shelf' or 'tf'
suffix on all ocn-glc coupling variables.
This allows testing with a MALI mesh where fjords are resolved.
The facemelting parameterization requires a subglacial runoff field as
an input.  Eventually, we may have this calculated prognostically in
E3SM from MALI and/or ELM, but that is not planned for the near-term.
Until then, a reasonable approximation is to use a constant historical
climatological field.

This commit updates the mpas.gis1to10kmR2 input file to include a
ismip6Runoff field provided by ISMIP6.  The field used is a 1995-2014
mean from the MIROC5 model, which was bias-corrected to match MAR over
that period.  See www.the-cryosphere.net/14/985/2020/ Fig. 2 and related
text for details.
This uses the layer that the desired depth is in, rather than the layer
above it.

Co-authored-by: Xylar Asay-Davis <[email protected]>
This commit adds an MPAS-Ocean namelist option for activating the
OCN-GLC TF coupling.  This option controls if E3SM should enable the
OCN-GLC TF coupling and also makes the associated TF calculations
conditional on the option being active.  The option defaults to false
and there currently are not any compsets that activate it, so it can
only be enabled with a namelist usermod at present.
Eventually, when we have compsets that require this mode, it should be
controlled analogously to MPASO_ISMF.
@matthewhoffman matthewhoffman added mpas-ocean mpas-albany-landice Coupler Stealth PR has feature which, if turned on, could change climate. fka FCC labels Sep 20, 2024
@matthewhoffman
Copy link
Contributor Author

Testing

Tested on Chrysalis with:

$ ./create_test --wait --walltime 0:30:00  ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling
Using project from config_machines.xml: e3sm
create_test will do up to 1 tasks simultaneously
create_test will use up to 160 cores simultaneously
Creating test directory /lcrc/group/e3sm/ac.mhoffman/scratch/chrys/ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling.20240920_132421_1imid2
RUNNING TESTS:
  ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling
Starting CREATE_NEWCASE for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling with 1 procs
Finished CREATE_NEWCASE for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling in 2.631567 seconds (PASS)
Starting XML for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling with 1 procs
Finished XML for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling in 0.406382 seconds (PASS)
Starting SETUP for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling with 1 procs
Finished SETUP for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling in 7.180280 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling with 1 procs
Finished SHAREDLIB_BUILD for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling in 192.547771 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling with 6 procs
Finished MODEL_BUILD for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling in 961.745927 seconds (PASS)
Starting RUN for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling with 1 proc on interactive node and 128 procs on compute nodes
Finished RUN for test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling in 5.035480 seconds (PEND). [COMPLETED 1 of 1]
Waiting for tests to finish
PASS ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling RUN
    Case dir: /lcrc/group/e3sm/ac.mhoffman/scratch/chrys/ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling.20240920_132421_1imid2
test-scheduler took 1250.1114015579224 seconds

Copy link

PR Preview Action v1.4.8
🚀 Deployed preview to https://E3SM-Project.github.io/E3SM/pr-preview/pr-6632/
on branch gh-pages at 2024-09-20 19:01 UTC

Copy link
Contributor

@xylar xylar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked over the ocean and coupler code, and am approving the code based on that inspection.

I ran the e3sm_cryo_developer suite on master:

./create_test --wait --walltime=1:00:00 "e3sm_cryo_developer" -g -b master_20240928 --baseline-root /lcrc/group/e3sm/ac.xylar/e3sm_baselines

and then on this branch after a test merge with master:

./create_test --wait --walltime=1:00:00 "e3sm_cryo_developer" -c -b master_20240928 --baseline-root /lcrc/group/e3sm/ac.xylar/e3sm_baselines

and saw the expected diffs (new output fields and namelist options):

FAIL ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel NLCOMP
FAIL ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel BASELINE master_20240928: FIELDLIST field lists differ (otherwise bit-for-bit)
FAIL ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel MEMCOMP [Errno 2] No such file or directory: '/lcrc/group/e3sm/ac.xylar/e3sm_baselines/master_20240928/ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel/cpl-mem.log'
FAIL ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel NLCOMP
FAIL ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel BASELINE master_20240928: FIELDLIST field lists differ (otherwise bit-for-bit)
FAIL ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel MEMCOMP [Errno 2] No such file or directory: '/lcrc/group/e3sm/ac.xylar/e3sm_baselines/master_20240928/ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel/cpl-mem.log'
FAIL PEM_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel NLCOMP
FAIL PEM_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel BASELINE master_20240928: FIELDLIST field lists differ (otherwise bit-for-bit)
FAIL PEM_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel MEMCOMP [Errno 2] No such file or directory: '/lcrc/group/e3sm/ac.xylar/e3sm_baselines/master_20240928/PEM_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel/cpl-mem.log'
FAIL PEM_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel NLCOMP
FAIL PEM_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel BASELINE master_20240928: FIELDLIST field lists differ (otherwise bit-for-bit)
FAIL PEM_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel MEMCOMP [Errno 2] No such file or directory: '/lcrc/group/e3sm/ac.xylar/e3sm_baselines/master_20240928/PEM_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel/cpl-mem.log'
FAIL PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel NLCOMP
FAIL PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel BASELINE master_20240928: FIELDLIST field lists differ (otherwise bit-for-bit)
FAIL PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel MEMCOMP [Errno 2] No such file or directory: '/lcrc/group/e3sm/ac.xylar/e3sm_baselines/master_20240928/PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.chrysalis_intel/cpl-mem.log'
FAIL PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel NLCOMP
FAIL PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel BASELINE master_20240928: FIELDLIST field lists differ (otherwise bit-for-bit)
FAIL PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel MEMCOMP [Errno 2] No such file or directory: '/lcrc/group/e3sm/ac.xylar/e3sm_baselines/master_20240928/PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel/cpl-mem.log'
FAIL PET_Ln5.T62_oQU240wLI.GMPAS-DIB-IAF-PISMF.chrysalis_intel TPUTCOMP Error: TPUTCOMP: Throughput changed by 5.85%: baseline=47.296 sypd, tolerance=5%, current=44.530 sypd
FAIL SMS_D_Ld1.TL319_IcoswISC30E3r5.GMPAS-JRA1p5-DIB-PISMF.chrysalis_intel.mpaso-jra_1958 NLCOMP
FAIL SMS_D_Ld1.TL319_IcoswISC30E3r5.GMPAS-JRA1p5-DIB-PISMF.chrysalis_intel.mpaso-jra_1958 BASELINE master_20240928: FIELDLIST field lists differ (otherwise bit-for-bit)
FAIL SMS_D_Ld1.TL319_IcoswISC30E3r5.GMPAS-JRA1p5-DIB-PISMF.chrysalis_intel.mpaso-jra_1958 MEMCOMP [Errno 2] No such file or directory: '/lcrc/group/e3sm/ac.xylar/e3sm_baselines/master_20240928/SMS_D_Ld1.TL319_IcoswISC30E3r5.GMPAS-JRA1p5-DIB-PISMF.chrysalis_intel.mpaso-jra_1958/cpl-mem.log'

<grid name="atm">TL319</grid>
<grid name="lnd">TL319</grid>
<grid name="ocnice">oQU240wLI</grid>
<grid name="rof">JRA025</grid>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this different from the r025 mesh used in high-res cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the mesh that the runoff for the JRA data forcing is on, but I don't know how it's different than our standard r025. @jonbob , do you know?

It looks like there are about 29 instances of this used for existing JRA grids.

Four new gridspecs were introduced recently that were brought in to this
branch after a rebase and were missed in the original renaming of
OCN2GLC_*MAPNAME to OCN2GLC_SHELF_*FMAPNAME.  This commit makes the
name change for those grids as well.
@jonbob jonbob added the BFB PR leaves answers BFB label Nov 5, 2024
Copy link
Contributor

@jonbob jonbob left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved based on visual inspection and testing

@jonbob
Copy link
Contributor

jonbob commented Nov 5, 2024

Testing shows:

  • new test ERS_Ld5.TL319_oQU240wLI_gis20.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glc_tf_coupling ran successfully
  • ERS.ne30_g16_rx1.A.chrysalis_intel showed NML and regular DIFF, as expected
  • ERP_Ld3.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.chrysalis_intel.allactive-pioroot1 showed NML DIFF, as expected

The last test also reported "BASELINE master: FIELDLIST field lists differ (otherwise bit-for-bit)" as a FAIL

@rljacob -- how do we document this kind of PR, if it's BFB but adding a field?

jonbob added a commit that referenced this pull request Nov 5, 2024
…(PR #6632)

OCN to GLC thermal forcing coupling

This pull request introduces coupling from OCN to GLC to pass thermal
forcing at a prescribed ocean depth (300 m) from MPAS-Ocean to MALI,
where MALI uses it to calculate grounded marine melting through an
existing 'facemelting' parameterization. Facemelting as a function of
ocean thermal state is a critical OCN/GLC coupling for Greenland, but it
is relevant to Antarctica as well. This OCN-GLC thermal forcing coupling
is activated whenever OCN is present and GLC is active.

This PR has pieces in MPAS-Ocean, the coupler, and MALI:
* In MPAS-Ocean, a new avgThermalForcingAtCritDepth field is added and
  calculated. The depth at which to calculate the thermal forcing is
  namelist configurable. The default is 300 m.
* In MALI, the field passed from the coupler is collected in
  ismip6_2dThermalForcing and used in the existing facemelting
  parameterization. Namelist and streams are updated to use facemelting
  in all simulations and output relevant variables. For compsets where
  TF is not being coupled, it will have values of 0 and result in no
  facemelting being applied; it is safe to have this option generally
  turned on.
* In the coupler, a new remapping object is created that handles the TF
  mapping independently of any other fields passed from OCN to GLC. New
  mapping files are added. Any coupling variables related to the
  existing ice-shelf coupling now include the 'shelf' suffix and all
  coupling variables related to this new thermal-forcing coupling have
  the 'tf' suffix.
* A new compset is created that is a standard G-case but with active
  MALI. New mapping files are added between the relevant grids for the
  new mapper.
* The new OCN2GLC_TF_SMAPNAME mapping should be a nearest source to
  destination mapping with the MPAS-Ocean source mesh masked to only
  include grid cells that are deeper than the critical depth to avoid
  extrapolating undefined values. A tool to generate that map file is
  MPAS-Tools
* Two new model_grids are added that are compatible with this feature
  and include the special mapping file required:
  TL319_IcoswISC30E3r5_gis1to10kmR2 and TL319_oQU240wLI_gis20
* The new coupling is controlled by a new namelist option in MPAS-Ocean:
  config_glc_thermal_forcing_coupling_mode. It defaults to 'off' and
  there are no compsets that explicitly set it for now, making this a
  stealth feature.

[NML] for new mappings, new ocean variable
[BFB] stealth feature (except A-cases due to new coupler field)

note: testing will report this as a FAIL due to:
"BASELINE master: FIELDLIST field lists differ (otherwise bit-for-bit)"
@rljacob
Copy link
Member

rljacob commented Nov 5, 2024

I thought adding a field makes the X-case diff. A-case is all data models. Why would that diff?

Also I never got an answer for how the JRA025 river grid is different from r025.

@jonbob
Copy link
Contributor

jonbob commented Nov 5, 2024

merged to next

@jonbob
Copy link
Contributor

jonbob commented Nov 5, 2024

@rljacob -- you're right about it being X-cases. The A-case was non-BFB due to the new field but not a baseline DIFF. I changed that in the PR description just now. As for the JRA025 vs r025, the JRA domain is defined from 0 to 360 longitude while it's -180 to +180 for r025. Otherwise I think they are very close to the same.

@jonbob
Copy link
Contributor

jonbob commented Nov 7, 2024

removed from next -- @rljacob force-pushed next to clear. Re-working to minimize impact on the coupler

@rljacob
Copy link
Member

rljacob commented Nov 21, 2024

Notes: still planning a rework

@matthewhoffman
Copy link
Contributor Author

@jonbob , the addition of the flds_tf logic makes sense to me, and I don't have any concerns about the details or the naming conventions. If you are happy with this solution as-is, I'm happy to proceed.

@jonbob
Copy link
Contributor

jonbob commented Dec 9, 2024

@matthewhoffman -- I'm not sure if this is intentional, but testing shows non-BFB behavior when MALI is active. It's apparently due to the change in the default setting for config_front_mass_bal_grounded.

@jonbob jonbob added non-BFB PR makes roundoff changes to answers. and removed BFB PR leaves answers BFB labels Dec 9, 2024
@matthewhoffman
Copy link
Contributor Author

@jonbob , yes, activating config_front_mass_bal_grounded was intentional - that is the option that lets MALI use the new ocean coupling to calculate submarine melt rates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Coupler mpas-albany-landice mpas-ocean NML non-BFB PR makes roundoff changes to answers. Stealth PR has feature which, if turned on, could change climate. fka FCC
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants