compressed *.npy for temp files

[JamiePringle]

Hi all --

I propose that the temporary *.npy files be compressed with the gzip module. This is a significant win for me, since a single year run for me produces a 13Tb temporary file if uncompressed, and a 4Tb file if compressed. For more modest sized projects the difference will be less important. However, I see no downside in compressing these files, since for most archetectures it reduces both the size of the files and the IO time. The changes are modest; the files are opened with gzip.open() when the files read and written.

The only downside is that any other code that reads these temporary files will need to be altered; but the effort in doing so is minor, as can be seen in the following diffs.

I have made and tested the changes with baseparticlefile.py and particlefilesoa.py. If this goes forward, the AOS file would have to be changed and tested.

The changes to baseparticlefile.py are

*** baseparticlefile.py_original        2022-05-08 21:31:10.097127456 -0400
--- baseparticlefile.py 2022-05-12 22:16:10.816457186 -0400
***************
*** 5,10 ****
--- 5,11 ----
  import string
  from abc import ABC
  from abc import abstractmethod
+ import gzip #cjmp
  
  import numpy as np
  
***************
*** 235,252 ****
          if not os.path.exists(self.tempwritedir):
              os.makedirs(self.tempwritedir)
  
          if len(data_dict) > 0:
!             tmpfilename = os.path.join(self.tempwritedir, str(len(self.file_list)) + ".npy")
!             with open(tmpfilename, 'wb') as f:
                  np.save(f, data_dict)
              self.file_list.append(tmpfilename)
  
          if len(data_dict_once) > 0:
!             tmpfilename = os.path.join(self.tempwritedir, str(len(self.file_list)) + '_once.npy')
!             with open(tmpfilename, 'wb') as f:
                  np.save(f, data_dict_once)
              self.file_list_once.append(tmpfilename)
! 
      @abstractmethod
      def get_pset_info_attributes(self):
          """
--- 236,255 ----
          if not os.path.exists(self.tempwritedir):
              os.makedirs(self.tempwritedir)
  
+         #cjmp 
          if len(data_dict) > 0:
!             tmpfilename = os.path.join(self.tempwritedir, str(len(self.file_list)) + ".npy.gz")
!             with gzip.open(tmpfilename, 'wb') as f:
                  np.save(f, data_dict)
              self.file_list.append(tmpfilename)
  
          if len(data_dict_once) > 0:
!             tmpfilename = os.path.join(self.tempwritedir, str(len(self.file_list)) + '_once.npy.gz')
!             with gzip.open(tmpfilename, 'wb') as f:
                  np.save(f, data_dict_once)
              self.file_list_once.append(tmpfilename)
!         #cjmp end
!             
      @abstractmethod
      def get_pset_info_attributes(self):
          """

and the changes to particlefilesoa.py are

*** particlefilesoa.py_original 2022-05-12 22:04:34.252482525 -0400
--- particlefilesoa.py  2022-05-12 22:11:25.488467565 -0400
***************
*** 3,8 ****
--- 3,9 ----
  from glob import glob
  import numpy as np
  import xarray as xr
+ import gzip
  
  try:
      from mpi4py import MPI
***************
*** 85,91 ****
          # loop over all files
          for npyfile in file_list:
              try:
!                 data_dict = np.load(npyfile, allow_pickle=True).item()
              except NameError:
                  raise RuntimeError('Cannot combine npy files into netcdf file because your ParticleFile is '
                                     'still open on interpreter shutdown.\nYou can use '
--- 86,95 ----
          # loop over all files
          for npyfile in file_list:
              try:
!                 #cjmp
!                 with gzip.open(npyfile,'rb') as f:
!                     data_dict = np.load(f, allow_pickle=True).item()
!                 #cjmp end
              except NameError:
                  raise RuntimeError('Cannot combine npy files into netcdf file because your ParticleFile is '
                                     'still open on interpreter shutdown.\nYou can use '
***************
*** 139,145 ****
              if os.path.exists(tempwritedir):
                  pset_info_local = np.load(os.path.join(tempwritedir, 'pset_info.npy'), allow_pickle=True).item()
                  for npyfile in pset_info_local['file_list']:
!                     tmp_dict = np.load(npyfile, allow_pickle=True).item()
                      for i in tmp_dict['id']:
                          if i in n_timesteps:
                              n_timesteps[i] += 1
--- 143,152 ----
              if os.path.exists(tempwritedir):
                  pset_info_local = np.load(os.path.join(tempwritedir, 'pset_info.npy'), allow_pickle=True).item()
                  for npyfile in pset_info_local['file_list']:
!                     #cjmp
!                     with gzip.open(npyfile,'rb') as f:
!                         tmp_dict = np.load(f, allow_pickle=True).item()
!                     #cjmp end
                      for i in tmp_dict['id']:
                          if i in n_timesteps:
                              n_timesteps[i] += 1

If this idea seems good to people, I could figure out github and make the changes to baseparticefile.py and particlefilesoa.py there.

Jamie

[erikvansebille]

Thanks for the info and idea, @JamiePringle. If you say that storing as zipped spy files does not really have downsides, then I think we can just give it a try. Can you create a Pull Request on GitHub?

[JamiePringle]

It is in pull request #1171 ; I hope it is trouble free.