Add e3sm_atm_nbfb comparison results to TestStatus.log

jhkennedy · jhkennedy · commit 65dc0242810a · 2019-08-27T18:00:57.000-04:00
EVV results will be reported to TestStatus.log which is what gets
displayed on CDASH instead of just to the test stdout
diff --git a/scripts/climate_reproducibility/README.md b/scripts/climate_reproducibility/README.md
@@ -1,5 +1,4 @@
-Climate reproducibility testing
-===============================
+# Climate reproducibility testing
 
 Requiring model changes to pass stringent tests before being accepted as part of E3SM’s main development 
 branch is critical for quickly and efficiently producing a trustworthy model.  Depending on their
@@ -43,8 +42,7 @@ whether or not non-bit-for-bit (nb4b) model changes are climate changing. The cu
  
  
  
-Running the tests
------------------
+## Running the tests
 
 These tests are built into E3SM-CIME as system tests and will be launched using the `create_test` scripts. 
 *However*, because these tests use high level statistics, they have additional python dependencies which
@@ -79,13 +77,13 @@ directory containing E3SM). Then to run one of the tests, you will use the `crea
 To run the `MVK` test and generate a baseline, you would run a command like: 
 
 ```
-./create_test MVK_PL.ne4_oQU240.FC5AV1C-04P2 -g --baseline-root "/PATH/TO/BASELINE" 
+./create_test MVK_PL.ne4_oQU240.FC5AV1C-L -g --baseline-root "/PATH/TO/BASELINE" 
 ```
 
 And to compare to the baseline, you would run a command like: 
 
 ```
-./create_test MVK_PL.ne4_oQU240.FC5AV1C-04P2 -c --baseline-root "/PATH/TO/BASELINE" 
+./create_test MVK_PL.ne4_oQU240.FC5AV1C-L -c --baseline-root "/PATH/TO/BASELINE" 
 ```
 
 *NOTE: The MVK run a 20 member ensemble for at least 13 months (using the last 12 for the 
@@ -95,34 +93,33 @@ option to `create_test`.*
 
 The full set of commands to run the MVK test used on Cori are:
 
-*Generate a baseline:*
+*Generate a baseline*
 ```
 cd $E3SM/cime/scripts
 
 source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_simple.sh
 
-./create_test MVK_PL.ne4_ne4.FC5AV1C-04P2 --baseline-root "${CSCRATCH}/baselines" --project acme -g -o --walltime 01:00:00
+./create_test MVK_PL.ne4_ne4.FC5AV1C-L --baseline-root "${CSCRATCH}/baselines" --project acme -g -o --walltime 01:00:00
 ```
 
-*Compare to a baseline:*
+*Compare to a baseline*
 ```
 cd $E3SM/cime/scripts
 
 source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_simple.sh
 
-./create_test MVK_PL.ne4_ne4.FC5AV1C-04P2 --baseline-root "${CSCRATCH}/baselines" --project acme -c --walltime 01:00:00
+./create_test MVK_PL.ne4_ne4.FC5AV1C-L --baseline-root "${CSCRATCH}/baselines" --project acme -c --walltime 01:00:00
 ```
 
-Test pass/fail and extended output
-----------------------------------
+## Test pass/fail and extended output
 
-When you launch these tests, CIME will ouput the location of the case directory, which will look 
+When you launch these tests and compare to a baseline, CIME will output the location of the case directory, which will look 
 something like this:
 
 ```
 # On cori-knl:
-./create_test MVK_PL.ne4_ne4.FC5AV1C-04P2 --baseline-root "${CSCRATCH}/baselines" --project acme -c --walltime 01:00:00
-    Creating test directory /global/cscratch1/sd/${USER}/acme_scratch/cori-knl/MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel.G.20190807_140111_rhfxn9
+./create_test MVK_PL.ne4_ne4.FC5AV1C-L --baseline-root "${CSCRATCH}/baselines" --project acme -c --walltime 01:00:00
+    Creating test directory /global/cscratch1/sd/${USER}/acme_scratch/cori-knl/MVK_PL.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID
 ```
 
 Let's call that directory `$CASE_DIR`. Once all the jobs are finished, navigate to that directory and 
@@ -132,83 +129,105 @@ you can `cat TestStatus` to determine if the test passed or failed by looking at
 cd $CASE_DIR
 cat TestStatus
     ...
-    PASS MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel BASELINE
+    PASS MVK_PL.ne4_ne4.FC5AV1C-L.cori-knl_intel BASELINE
     ...
 
 ```
 
-To get some basic summary statistics about the test that was run, look in the output of the final job 
-submission for EVV's analysis:
+To get some basic summary statistics about the test that was run, look in the `TestStatus.log` file:
 
- ```
-view MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel.G.20190807_140111_rhfxn9.C.YYYYMMDD_HHMMSS_RANDOMID.test.oJOBID
-    ...
-    2019-08-14 22:09:02 CASE.RUN HAS FINISHED 
-    -------------------------------------------------------------------- 
-                       ______  __      __ __      __                     
-                      |  ____| \ \    / / \ \    / /                     
-                      | |__     \ \  / /   \ \  / /                      
-                      |  __|     \ \/ /     \ \/ /                       
-                      | |____     \  /       \  /                        
-                      |______|     \/         \/                         
-                                                                         
-        Extended Verification and Validation for Earth System Models     
-    -------------------------------------------------------------------- 
-     
-      Current run: 2019-08-14 21:51:32 
-      User: ${USER} 
-      OS Type: Linux 4.12.14-25.22_5.0.70-cray_ari_c 
-      Machine: nid06598 
-       
-    ------------------------------------------------------------------- 
-     ----------------------------------------------------------------- 
-       Beginning extensions test suite  
-     -----------------------------------------------------------------
-
-     
-        Kolmogorov-Smirnov test: YYYYMMDD_HHMMSS_RANDOMID 
-          Variables analyzed: 378 
-          Rejecting: 9 
-          Critical value: 13.0 
-          Ensembles: identical 
-     
-     ----------------------------------------------------------------- 
-       Extensions test suite complete  
-     ----------------------------------------------------------------- 
-     
-    ------------------------------------------------------------------- 
-     Done!  Results can be seen in a web browser at: 
-       /global/cscratch1/sd/${USER}/acme_scratch/cori-knl/MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel.G.20190807_140111_rhfxn9/run/MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel.G.20190807_140111_rhfxn9.evv/index.html 
-    -------------------------------------------------------------------
-    ...
+```
+2019-08-14  22:09:02: BASELINE PASS for test 'YYYYMMDD_HHMMSS_RANDOMID'. 
+    Case: YYYYMMDD_HHMMSS_RANDOMID; Test status: pass; Variables analyzed: 118; Rejecting: 0; Critical value: 13; Ensembles: statistically identical 
+    EVV results can be viewed at: /global/cscratch1/sd/${USER}/acme_scratch/cori-knl/MVK_PL.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID/run/MVK_PL.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID.evv/ 
+    EVV viewing instructions can be found at: https://github.com/E3SM-Project/E3SM/blob/master/cime/scripts/climate_reproducibility/README.md#test-passfail-and-extended-output 
 ```
 
-EVV also prints the location of the output website where you can see the details of the analysis. For 
+EVV reports the location of the output website where you can see the details of the analysis. For 
 the MVK test, you will be able to view per variable Q-Q plots, P-P plots, the K-S test statistic, and 
 whether it rejects or accepts the null hypothesis, as well as a description of the test itself -- you 
 can see an example of the output website [here](http://livvkit.github.io/evv4esm/).
 
-Please note: the output website uses some JavaScript to render elements of the page (especially figures), 
-and opening up the `index.html` file using the `file://` protocol in a web browser will likely not work 
-well (most browser have stopped allowing access to "local resources" like JavaScript through the `file://` 
-protocol). You can view the website by either copying it to a hosted location (`~/WWW` which is hosted at 
-`http://users.nccs.gov/~user` on Titan, for example) or copying it to your local machine and running a 
-local http server (included in python!) and viewing it through an address like `http://0.0.0.0:8000/index.html`.  
+To view the website, you can either tunnel the website to your local machine through ssh, or copy 
+the website directory to your machine and view it using EVV. 
+
+### View via ssh
+
+For this example, we'll assume the tests were run on Cori at NERSC, but these instructions should be
+easily adaptable to any E3SM supported machine. First, log into Cori via ssh and connect your local 
+8080 port to the 8080 port on Cori: 
+
+```
+ssh -L 8080:localhost:8080 [USER]@cori.nersc.gov 
+```
+
+Activate the `e3sm_simple` environment:
+
+```
+source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_simple.sh
+```
+
+Navigate to the case's run directory:
+
+```
+pushd ${CASE_DIR}/run
+```
+
+Then, using EVV, serve the website over port 8080:
 
-**For the easiest viewing** we recommend copying the website to your local machine, and using EVV to 
-view it. you can install EVV locally by running this command:
+```
+evv -o PGN_P1x1.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID.evv -s 8080
+```
+
+Evv will then report to you the URL where you can view the website:
+
+```
+
+--------------------------------------------------------------------
+                   ______  __      __ __      __                    
+                  |  ____| \ \    / / \ \    / /                    
+                  | |__     \ \  / /   \ \  / /                     
+                  |  __|     \ \/ /     \ \/ /                      
+                  | |____     \  /       \  /                       
+                  |______|     \/         \/                        
+                                                                    
+    Extended Verification and Validation for Earth System Models    
+--------------------------------------------------------------------
+
+  Current run: 2019-08-27 14:16:49
+  User: kennedyj
+  OS Type: Linux 4.12.14-150.27-default
+  Machine: cori07
+  
+
+Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/)
+
+View the generated website by navigating to:
+
+    http://0.0.0.0:8080/PGN_P1x1.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID.evv/index.html
+
+Exit by pressing `ctrl+c` to send a keyboard interrupt.
+```
+
+You can now either click that link or copy-paste that link into your favorite web 
+browser to view the output website.
+
+### View a local copy
+
+For this example, we'll assume the tests were run on Cori at NERSC, but these instructions should be
+easily adaptable to any E3SM supported machine. Install `e3sm_simple` locally and activate it:
 
 ```
-conda install evv4esm
+conda create -n e3sm_simple -c conda-forge -c e3sm e3sm-simple
+conda activate e3sm_simple
 ``` 
 
 Then, copy the website to your local machine, and view it: 
 
-
 ```
 # on your local machine
-scp -r /lustre/atlas/proj-shared/cli115/$USER/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID/run/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.eve . 
-evv -o MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.evv -s
+scp -r /global/cscratch1/sd/${USER}/acme_scratch/cori-knl/MVK_PL.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID/run/MVK_PL.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID.evv . 
+evv -o MVK_PL.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID.evv -s
     --------------------------------------------------------------------
                        ______  __      __ __      __                    
                       |  ____| \ \    / / \ \    / /                    
@@ -230,9 +249,19 @@ evv -o MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.evv -
     
     View the generated website by navigating to:
     
-        http://0.0.0.0:8000/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.evv/index.html
+        http://0.0.0.0:8000/MVK_PL.ne4_ne4.FC5AV1C-L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID.evv/index.html
     
     Exit by pressing `ctrl+c` to send a keyboard interrupt.
     
 ```
 
+You can now either click that link or copy-paste that link into your favorite web 
+browser to view the output website.
+
+
+**Please note:** the output website uses some JavaScript to render elements of the page (especially figures), 
+and opening up the `index.html` file using the `file://` protocol in a web browser will likely not work 
+well (most browser have stopped allowing access to "local resources" like JavaScript through the `file://` 
+protocol). You can view the website by either copying it to a hosted location (`~/WWW` which is hosted at 
+`http://users.nccs.gov/~user` on Titan, for example) or copying it to your local machine and running a 
+local http server (included in python!) and viewing it through an address like `http://0.0.0.0:8000/index.html`.  
diff --git a/scripts/lib/CIME/SystemTests/mvk.py b/scripts/lib/CIME/SystemTests/mvk.py
@@ -14,7 +14,7 @@
 from CIME.SystemTests.system_tests_common import SystemTestsCommon
 from CIME.case.case_setup import case_setup
 from CIME.hist_utils import _get_all_hist_files
-from CIME.utils import safe_copy, SharedArea
+from CIME.utils import safe_copy, SharedArea, append_testlog
 
 import evv4esm  # pylint: disable=import-error
 from evv4esm.__main__ import main as evv  # pylint: disable=import-error
@@ -133,10 +133,24 @@ def _compare_baseline(self):
             with open(os.path.join(evv_out_dir, 'index.json')) as evv_f:
                 evv_status = json.load(evv_f)
 
+            comments = ""
             for evv_elem in evv_status['Data']['Elements']:
                 if evv_elem['Type'] == 'ValSummary' \
                         and evv_elem['TableTitle'] == 'Kolmogorov-Smirnov test':
+                    comments = "; ".join("{}: {}".format(key, val) for key, val
+                                         in evv_elem['Data'][test_name][''].items())
                     if evv_elem['Data'][test_name]['']['Test status'].lower() == 'pass':
                         self._test_status.set_status(CIME.test_status.BASELINE_PHASE,
                                                      CIME.test_status.TEST_PASS_STATUS)
-                        break
+                    break
+
+                status = self._test_status.get_status(CIME.test_status.BASELINE_PHASE)
+                comments = "{} {} for test '{}'.\n" \
+                           "    {}\n" \
+                           "    EVV results can be viewed at: {}\n" \
+                           "    EVV viewing instructions can be found at: " \
+                           "        https://github.com/E3SM-Project/E3SM/blob/master/cime/scripts/" \
+                           "climate_reproducibility/README.md#test-passfail-and-extended-output" \
+                           "".format(CIME.test_status.BASELINE_PHASE, status, test_name, comments, evv_out_dir)
+
+                append_testlog(comments, self._orig_caseroot)
diff --git a/scripts/lib/CIME/SystemTests/pgn.py b/scripts/lib/CIME/SystemTests/pgn.py
@@ -22,7 +22,7 @@
 import CIME.test_status
 from CIME.SystemTests.system_tests_common import SystemTestsCommon
 from CIME.case.case_setup import case_setup
-from CIME.utils import expect
+from CIME.utils import expect, append_testlog
 
 import evv4esm  # pylint: disable=import-error
 from evv4esm.extensions import pg  # pylint: disable=import-error
@@ -173,13 +173,27 @@ def _compare_baseline(self):
             with open(os.path.join(evv_out_dir, 'index.json'), 'r') as evv_f:
                 evv_status = json.load(evv_f)
 
+            comments = ""
             for evv_elem in evv_status['Data']['Elements']:
                 if evv_elem['Type'] == 'ValSummary' \
                         and evv_elem['TableTitle'] == 'Perturbation growth test':
+                    comments = "; ".join("{}: {}".format(key, val) for key, val
+                                         in evv_elem['Data'][test_name][''].items())
                     if evv_elem['Data'][test_name]['']['Test status'].lower() == 'pass':
                         self._test_status.set_status(CIME.test_status.BASELINE_PHASE,
                                                      CIME.test_status.TEST_PASS_STATUS)
-                        break
+                    break
+
+            status = self._test_status.get_status(CIME.test_status.BASELINE_PHASE)
+            comments = "{} {} for test '{}'.\n" \
+                       "    {}\n" \
+                       "    EVV results can be viewed at: {}\n" \
+                       "    EVV viewing instructions can be found at: " \
+                       "        https://github.com/E3SM-Project/E3SM/blob/master/cime/scripts/" \
+                       "climate_reproducibility/README.md#test-passfail-and-extended-output" \
+                       "".format(CIME.test_status.BASELINE_PHASE, status, test_name, comments, evv_out_dir)
+
+            append_testlog(comments, self._orig_caseroot)
 
     def run_phase(self):
         logger.debug("PGN_INFO: RUN PHASE")
diff --git a/scripts/lib/CIME/SystemTests/tsc.py b/scripts/lib/CIME/SystemTests/tsc.py
@@ -15,7 +15,7 @@
 from CIME.SystemTests.system_tests_common import SystemTestsCommon
 from CIME.case.case_setup import case_setup
 from CIME.hist_utils import rename_all_hist_files, _get_all_hist_files
-from CIME.utils import safe_copy, SharedArea
+from CIME.utils import safe_copy, SharedArea, append_testlog
 
 import evv4esm  # pylint: disable=import-error
 from evv4esm.__main__ import main as evv  # pylint: disable=import-error
@@ -151,13 +151,27 @@ def _compare_baseline(self):
             with open(os.path.join(evv_out_dir, 'index.json'), 'r') as evv_f:
                 evv_status = json.load(evv_f)
 
+            comments = ""
             for evv_elem in evv_status['Data']['Elements']:
                 if evv_elem['Type'] == 'ValSummary' \
                         and evv_elem['TableTitle'] == 'Time step convergence test':
+                    comments = "; ".join("{}: {}".format(key, val) for key, val
+                                         in evv_elem['Data'][test_name][''].items())
                     if evv_elem['Data'][test_name]['']['Test status'].lower() == 'pass':
                         self._test_status.set_status(CIME.test_status.BASELINE_PHASE,
                                                      CIME.test_status.TEST_PASS_STATUS)
-                        break
+                    break
+
+            status = self._test_status.get_status(CIME.test_status.BASELINE_PHASE)
+            comments = "{} {} for test '{}'.\n" \
+                       "    {}\n" \
+                       "    EVV results can be viewed at: {}\n" \
+                       "    EVV viewing instructions can be found at: " \
+                       "        https://github.com/E3SM-Project/E3SM/blob/master/cime/scripts/" \
+                       "climate_reproducibility/README.md#test-passfail-and-extended-output" \
+                       "".format(CIME.test_status.BASELINE_PHASE, status, test_name, comments, evv_out_dir)
+
+            append_testlog(comments, self._orig_caseroot)
 
     def _generate_baseline(self):
         super(TSC, self)._generate_baseline()
