Develope obsForge utilities

[ilianagenkova]

Develop a number of obsForge utilities for the IODA (netcdf and odb/odc formatted) observation files:

A) Develop tools that mimic the functionality of the NCEPLIB-bufr tools:
https://github.com/NOAA-EMC/NCEPLIBS-bufr/tree/develop/utils
A1) sinv - print an inventory of unique satellite IDs (along with their names in characters) found in the IODA files, and the number of observation from each satellite
A2) binv - print inventory of tanks which populated the bufr dump and now the IODA files, and the number of observation from each tank
A3) split_by_subset - split the IODA files to subsets, which represent observation from a single tank
A4) (no analogos tool for bufr) - write a tool that can exract an observation based on it's index or range of indices
A5) debufr - write out the IODA file as ascii table; provide configuration option to write out only certain mnemonics/variables

B) Develop a tool that mimmic the current go_chkdat and minv tools, i.e. extract the earliest and latest observation times from the IODA file

C) Develop reporting of observation counts - daily and monthy, similar to the "status" files from the legacy obsproc, for use by NCO and plotting on the Real Time Data Monitoring (RTDM) website. Coordinate with NCO on the file formatting (the current reports are parsed by grep/sed/awk)

Notes:

1. obsForge in GFS v17 will not preserve tank info in the IODA files. We will lean on the bufr dump at times for diagnostics
   obsForge in GFS v18 (JEDI) will incorporate tank info in the IODA files - A2) and A3) need to wait until that time.
2. A and B are smaller stand alone tools. C, on the other hand, is a larger effort that needs to be coordinated with NCO/SDM at some point.

Update: 1)Dan requested that the tools are capable of handling IODA netcdf and ODB/ODC data formats https://github.com/ecmwf/odc
2)Use the APIs Ron has developed to authomaticlaly recognize and read the obs files (wether it's bufr, netcdf, odb) and generate counts for the report

[CoryMartin-NOAA]

I've started something like C here: https://github.com/NOAA-EMC/DA-utils/tree/develop/src/ioda-dump but it is definitely a work in progress

[ilianagenkova]

I am getting input from Dan now, so stay tuned for updates.

Update: Just had a meeting with Dan, the scope of the task expanded, and I updated the description. I am not familiar with the APIs so perhaps we should meet and coordinate before Hyundeok starts. Plan before execute.

[rmclaren]

Just FYI, some of these things could probably be accomplished by using ncdump utility.

[CoryMartin-NOAA]

Utilities like ncdump will work for netCDF backend files but I think we would have to utilize the IODA C++ API to ensure interoperability between the different backends?

[ilianagenkova]

@CoryMartin-NOAA , would @HyundeokChoi-NOAA be PR-ing his utilities to obsForge repo or da-utils repo?

I just met with Guillaume and Ed, to discuss obs monitoring in JEDI. I got some ideas how Ed and Hyundeok work may overlap/leverage. Let's talk on Wednesday

[CoryMartin-NOAA]

@ilianagenkova probably da-utils so that they can be built/used by DA developers.

And okay, sounds good to me.