Remove `DataModel.history`

[braingram]

It only has a few uses in wfs_combine (see spacetelescope/jwst#9744):

        new_model.history.append(f"Flip dithers = {self.flip_dithers}")
        new_model.history.append(f"WFS_COMBINE refine offset = {self.do_refine}")
        new_model.history.append(
            f"WFS_COMBINE X offset applied {str(self.off_x)} pixels "
            f"actual offset {str(round(self.flt_off_x, 2))} pixels"
        )
        new_model.history.append(
            f"WFS_COMBINE Y offset applied {str(self.off_y)} pixels "
            f"actual offset {str(round(self.flt_off_y, 2))} pixels"
        )

All of these may be more useful as log messages since the log messages are also recorded to FITS or ASDF file when an datamodel is saved and even better the log messages are shown to the terminal (the history appends are not).

"under the hood" DataModel.history modifies AsdfFile.tree["history"].

>> m.history.append("foo")
>> m._asdf.tree["history"]
{'entries': [{'description': 'foo',
   'time': datetime.datetime(2025, 8, 13, 14, 47, 58, tzinfo=datetime.timezone.utc)}]}

and there is already stable API in asdf for modifying the history via AsdfFile.add_history_entry:

>> m._asdf.add_history_entry("foo")  # where m is a new model 
{'entries': [{'description': 'foo',
   'time': datetime.datetime(2025, 8, 13, 14, 49, 21, tzinfo=datetime.timezone.utc)}]}

I propose that we:

• update wfs_combine to use log messages
• wrap AsdfFile.add_history_entry
• deprecate DataModel.history (and mention that after removal Datamodel.history.entries will return the history entries via the standard ASDF-to-attribute mapping of all other metadata)

[melanieclarke]

DataModel.history may be used for reference file generation -- better check with INS.

Would your suggested change to wrap add_history_entry still update the FITS HISTORY comments or would it only exist in the ASDF tree?

[braingram]

Would your suggested change to wrap add_history_entry still update the FITS HISTORY comments or would it only exist in the ASDF tree?

Thanks. Yeah add_history_entry is the same as history.append(...). In both cases the history portion of the ASDF tree is modified which stdatamodels then converts to HISTORY cards on write (and read).

[braingram]

DataModel.history may be used for reference file generation -- better check with INS.

How do you suggest to go about this?

[melanieclarke]

Let's ping @drlaw1558 and @hayescr.

David, Chris, the proposal here is to deprecate the datamodel history attribute, and support adding history messages via an add_history_entry function instead. The jwst pipeline in general does not add history messages, but I think it's possible reference file procedures might be using it.

In particular, I know that when I wrote the code for generating some of the NIRSpec reference files, I added history messages via calls to this attribute, like model.history.append(message). I don't know if that's still in use, or if it would be a problem to move to a different method of adding history messages to reference FITS files.

[nden]

I've seen INS scripts generating reference files that use it. It's common to add HISTORY FITS keywords to reference or science files and these are not considered equivalent to the log messages. I am not sure what was meant in wfs_combine. This task is not used by the pipeline, it was written for wavefront sensing.
That said perhaps a long enough deprecation process may be sufficient after informing INS though it may be also annoying to them.

[braingram]

Thanks all. To clarify, with #562 both DataModel.history.append and DataModel.add_history_entry add a history entry that when saved to a FITS file produces a HISTORY card in the primary header.

add_history_entry has a few benefits:

• it's standard API for asdf
• it will allow removal of the history submodule and HistoryList class (which has a hard reference to the AsdfFile instance which increases the risk of a circular reference that can result in DataModel instances not being garbage collected)
• since it uses the underlying history key in the AsdfFile instance it matches the output from DataModel.info (DataModel.info reports entries at history.entries, attempts to access this result in an AttributeError)

The one benefit I see with keeping HistoryList is that's it's consistent with old code. Are there other benefits?

[nden]

@braingram I don't see any other benefits for keeping HistoryList. Thanks for the explanation. 👍 on removing it after a deprecation period.

[hayescr]

Thanks for the ping @melanieclarke. We can poll the INS teams and see how many of them are using the DataModel.history.append functionality and whether they would foresee any issue with moving to DataModel.add_history_entry (I can confirm at least on the NIRSpec side we still use this functionality for updating reference file histories). Presumably if the new method functions the same and INS teams are only using the to record qualitative information in the DataModel.history (e.g., for reference file generation) it would hopefully be an easy conversion, but some lead time with a deprecation period would help.

Would the DataModel.add_history_entry be added before removing DataModel.history (which might allow some overlap time for migrating to the new method) or do you expect this to be done at the same time?

[braingram]

Thanks for the ping @melanieclarke. We can poll the INS teams and see how many of them are using the DataModel.history.append functionality and whether they would foresee any issue with moving to DataModel.add_history_entry (I can confirm at least on the NIRSpec side we still use this functionality for updating reference file histories). Presumably if the new method functions the same and INS teams are only using the to record qualitative information in the DataModel.history (e.g., for reference file generation) it would hopefully be an easy conversion, but some lead time with a deprecation period would help.

Thanks! These details are helpful.

Would the DataModel.add_history_entry be added before removing DataModel.history (which might allow some overlap time for migrating to the new method) or do you expect this to be done at the same time?

The current plan is to add DataModel.add_history_entry and add a deprecation warning for DataModel.history at the same time (but DataModel.history will continue to work until removed). The deprecation warning won't by default cause failures. We can stagger those changes if it's helpful. If you're curious feel free to test out the code at:
#562
installable with:
pip install "git+https://github.com/braingram/stdatamodels.git@deprecate_history"

[drlaw1558]

Hm, model.history is used pretty extensively by the MIRI reference file creation codes. It could be updated, but doing so would be a significant effort for multiple different codes. What's the motivation for wanting to remove this function?

[braingram]

Hm, model.history is used pretty extensively by the MIRI reference file creation codes. It could be updated, but doing so would be a significant effort for multiple different codes. What's the motivation for wanting to remove this function?

Thanks for looking at this.

To recap the motivation here is that HistoryList is a less-maintained, non-standard (and buggy) version of add_history_entry. Both record to HISTORY cards when written to a FITS file. Currently model.history is a special property that returns a HistoryList which shadows the actual file history and returns something that is inconsistent with the model.info output.

[braingram]

@drlaw1558 I'd also be happy to update any code using model.history if you can point me to where it's located.

[braingram]

Also the current history handling hides useful information about a file. Since model.history prevents accessing the asdf tree "history" key it does not allow a user to inspect the versions of packages used to produce the file (which is recorded under the asdf "history" key). For example a random regtest file "dev/nirspec/imaging/usf_assign_wcs.fits" returns an empty list for m.history which does not include the following information that is under the "history" key:

{'extensions': [{'extension_class': 'asdf.extension.BuiltinExtension',
   'software': {'name': 'asdf', 'version': '2.7.0'}},
  {'extension_class': 'astropy.io.misc.asdf.extension.AstropyAsdfExtension',
   'software': {'name': 'astropy', 'version': '4.0.1.post1'}},
  {'extension_class': 'gwcs.extension.GWCSExtension',
   'software': {'name': 'gwcs', 'version': '0.13.0'}},
  {'extension_class': 'astropy.io.misc.asdf.extension.AstropyExtension',
   'software': {'name': 'astropy', 'version': '4.0.1.post1'}},
  {'extension_class': 'jwst.transforms.jwextension.JWSTExtension',
   'software': {'name': 'jwst', 'version': '0.16.3.dev82+g6bf758be'}}]}