Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Look into recovering missing record-level HDF5 Attributes in raw data files that are not cleanly closed #110

Open
bieryAtFnal opened this issue Dec 16, 2024 · 0 comments

Comments

@bieryAtFnal
Copy link
Collaborator

In October/November 2024, there were several PDS calibration runs that did not end gracefully, and this resulted in raw data files that were not cleanly closed.

A utility was developed to recover missing file-level HDF5 Attributes in these files and thereby allow successful creation of the file-transfer metadata and successful transfer of the raw data files to offline storage. That utility is described here.

As part of reviewing that PR, Wes noticed that there may be additional recovery steps that we can take if/when we see similar un-closed raw data files in the future. We may be able to recover TriggerRecord-level HDF5 Attributes for the last TriggerRecord in un-closed files, and thereby make those records read-able offline.

For reference, the hdf5_dump.py utility was successfully run on all of the TRs in one of the sample unclosed PDS raw data files. This seems to imply that all of the data is available for the last TR in the file, and it may just be missing record-level Attributes that prevents the last TR from being examined by more sophisticated tools.

This Issue suggests that we look into all of this at some point in time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant