implementation of asynchronous file-I/O

[lars2015]

Is your feature request related to a problem?

• a significant amount of runtime of the simulations is required for reading meteo data (in particular for ERA5 data)

Describe the solution you'd like

• implement asynchronous file-I/O so that meteo data for the next time interval are being read ahead of time while trajectories calculations for the current time interval are conducted in parallel

Describe alternatives you've considered

• compression of meteo data to reduce waiting time for file-I/O

Additional context

• Example of runtime measurements of a recent GPU run on JUWELS Booster with 100 million particles and using ERA5 data (packed binary):

TIMER_READ_MET_BIN = 35.306 s    (min= 1.09297 s, mean= 1.41225 s, max= 2.2265 s, n= 25)
TIMER_GROUP_INIT = 0.553 s
TIMER_GROUP_MEMORY = 19.416 s
TIMER_GROUP_INPUT = 36.961 s
TIMER_GROUP_PHYSICS = 89.866 s
TIMER_GROUP_OUTPUT = 32.075 s
TIMER_TOTAL = 178.871 s

• Reading the ERA5 requires about 20% of the total runtime.

[lars2015]

Implementation of async-I/O by Kaveh:

• Kaveh added new code in libtrac.c and trac.c to enable async I/O for the meteo files
• code requires four instead of two meteo time steps to be kept in memory
• the new code can be enabled with the ASYNCIO flag in the Makefile
• first tests indicate that trajectory calculations show different results when ASYNCIO is enabled
• code is considered experimental and is switched off by default for now
• TODO: further testing/debugging is required

[lars2015]

Implementation of MET_CACHE flag:

• try loading meteo files into the disk cache as a background job:
  • https://serverfault.com/questions/406308/tell-the-linux-kernel-to-put-a-file-in-the-disk-cache
  • https://unix.stackexchange.com/questions/156166/file-access-time-after-loading-file-into-the-cache
• implemented a control parameter MET_CACHE in libtrac.c which uses system calls from C to trigger loading the next meteo file into the disk cache:

    /* Caching... */
    if (ctl->met_cache && t != ctl->t_stop) {
      get_met_help(ctl, t + 1.1 * ctl->dt_met * ctl->direction,
		   ctl->direction, ctl->metbase, ctl->dt_met, cachefile);
      sprintf(cmd, "cat %s > /dev/null &", cachefile);
      LOG(1, "Caching: %s", cachefile);
      if (system(cmd) != 0)
	WARN("Caching command failed!");
    }

• test on JUWELS Booster with ERA5 data:
  • flag on: TIMER_READ_MET_BIN = 36.268 s
  • flag off: TIMER_READ_MET_BIN = 41.463 s
  • small benefit from this (runtime reduced by ~12%)
• TODO:
  • more detailed testing for CPU and GPU jobs to see when/where the method works
  • does it matter whether we use "cat" or "tail" to load to disk cache?
  • find a smarter way to load files to the disk cache? (mmap?)

[lars2015]

Found this tool which allows for portable file system cache diagnostics and control:

• vmtouch - the Virtual Memory Toucher
• https://hoytech.com/vmtouch/