You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It has been noticed that the creation of the JSON metadata files on np04-srv-001 and -002 often takes a non-trivial amount of time per file.
This is because we need to open up the raw data file to fetch information that we put in the JSON metadata file. The worst case is building up the list of TriggerRecord numbers in the file (~5 seconds when there are no "write" operations going on), but even just opening the file to read an attribute can take a second or two if the file is not cached.
One option is to talk with our offline colleagues and see if they really need all of the JSON metadata elements that we are now providing, and if now, drop them.
Another option might be to have the HDF5-file-writing code create a small file that accompanies each raw data file, and that companion file has the information that is needed by the JSON metadata file creator and whatever else may want to know about the contents of the raw data file.
This latter option isn't a totally new idea. We have talked about companion files in the past that tell us the full list of files that contain a particular trigger, when a trigger spans multiple files, and what I'm suggestion above might be a natural part of that.
The text was updated successfully, but these errors were encountered:
It has been noticed that the creation of the JSON metadata files on np04-srv-001 and -002 often takes a non-trivial amount of time per file.
This is because we need to open up the raw data file to fetch information that we put in the JSON metadata file. The worst case is building up the list of TriggerRecord numbers in the file (~5 seconds when there are no "write" operations going on), but even just opening the file to read an attribute can take a second or two if the file is not cached.
One option is to talk with our offline colleagues and see if they really need all of the JSON metadata elements that we are now providing, and if now, drop them.
Another option might be to have the HDF5-file-writing code create a small file that accompanies each raw data file, and that companion file has the information that is needed by the JSON metadata file creator and whatever else may want to know about the contents of the raw data file.
This latter option isn't a totally new idea. We have talked about companion files in the past that tell us the full list of files that contain a particular trigger, when a trigger spans multiple files, and what I'm suggestion above might be a natural part of that.
The text was updated successfully, but these errors were encountered: