Move common resample code to stcal #8986
Conversation
mcara
force-pushed
the
new-resample-cls2
branch
from
adc3ca4 to
c7286ee
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8986 +/- ##
==========================================
- Coverage 76.81% 76.64% -0.18%
==========================================
Files 496 496
Lines 45610 45421 -189
==========================================
- Hits 35034 34811 -223
- Misses 10576 10610 +34
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Thanks for putting this together.
I'd like to see the CI run and regression tests for this PR. I added one comment about psutil which should fix one of the issues preventing the CI from running.
Would you update the pyproject.toml to point to the source branch of spacetelescope/stcal#320? That should (hopefully) allow the CI to run so we can see the coverage and test results with this PR.
Also would you queue up a regtest run?
|
Regression tests pass: https://github.com/spacetelescope/RegressionTests/actions/runs/12059744878 |
jwst/resample/resample_utils.py
Outdated
There was a problem hiding this comment.
Same question here: weren't a lot of the functions in here moved to stcal, so shouldn't they be removed from here?
mcara
force-pushed
the
new-resample-cls2
branch
from
3df4da3 to
3553412
Compare
mcara
force-pushed
the
new-resample-cls2
branch
from
1db5ba1 to
679aa51
Compare
|
Initial regression tests that were using the new code only for imaging are: https://github.com/spacetelescope/RegressionTests/actions/runs/12146883313 After switching spectral code to use the new class, here are the results of regression tests: https://github.com/spacetelescope/RegressionTests/actions/runs/12155412170 I most failures that I see are unrelated to my code, I think (such as |
There was a problem hiding this comment.
This is looking great Mihai! Thanks for all the hard work. Most of these comments are minor.
It still feels a bit sad to have to convert the datamodels into dictionaries to pass into stcal, but I can't really think of a better way to do it while maintaining the observatory-agnostic nature of stcal. So I guess I'm happy with it. Maybe it's a bit easier on the romancal side when working with asdf files as compared with fits?
| # delete these to reduce the number of failures in the regression tests | ||
| # TODO: this is helpful info that should be stored in products. | ||
| # del model.meta.resample.pixel_scale_ratio | ||
| # del model.meta.resample.pixfrac | ||
| # del model.meta.resample.weight_type | ||
| # del model.meta.resample.pointings | ||
| # del model.meta.cal_step.resample |
There was a problem hiding this comment.
Can you elaborate on why these cause failures? should these be reinstated before merge?
There was a problem hiding this comment.
It seems like currently for spectral data none of the above fields related to resample info are being set. So, if I turn this ON right now, you will see multiple failures in the regression tests. I think these should be turned on before merging, if @melanieclarke is not against storing this info for spectral data.
There was a problem hiding this comment.
I am all for storing this info for spectral data! I thought these were added, though, in the ResampleSpecStep? There may be some weirdness around what happens with MultiSlitModels.
mcara
force-pushed
the
new-resample-cls2
branch
from
bb0839e to
aff1ded
Compare
Closes #8695
This PR add the common resample code used by both JWST and Roman pipelines to stcal. For now only imaging mode was switched to the new code. This PR supersedes #8695
This work is related to https://jira.stsci.edu/browse/AL-835
The code in this PR requires the code from spacetelescope/stcal#320
Tasks
Build 11.3(use the latest build if not sure)no-changelog-entry-needed)changes/:echo "changed something" > changes/<PR#>.<changetype>.rst(see below for change types)docs/pageokify_regteststo update the truth filesnews fragment change types...
changes/<PR#>.general.rst: infrastructure or miscellaneous changechanges/<PR#>.docs.rstchanges/<PR#>.stpipe.rstchanges/<PR#>.datamodels.rstchanges/<PR#>.scripts.rstchanges/<PR#>.fits_generator.rstchanges/<PR#>.set_telescope_pointing.rstchanges/<PR#>.pipeline.rststage 1
changes/<PR#>.group_scale.rstchanges/<PR#>.dq_init.rstchanges/<PR#>.emicorr.rstchanges/<PR#>.saturation.rstchanges/<PR#>.ipc.rstchanges/<PR#>.firstframe.rstchanges/<PR#>.lastframe.rstchanges/<PR#>.reset.rstchanges/<PR#>.superbias.rstchanges/<PR#>.refpix.rstchanges/<PR#>.linearity.rstchanges/<PR#>.rscd.rstchanges/<PR#>.persistence.rstchanges/<PR#>.dark_current.rstchanges/<PR#>.charge_migration.rstchanges/<PR#>.jump.rstchanges/<PR#>.clean_flicker_noise.rstchanges/<PR#>.ramp_fitting.rstchanges/<PR#>.gain_scale.rststage 2
changes/<PR#>.assign_wcs.rstchanges/<PR#>.badpix_selfcal.rstchanges/<PR#>.msaflagopen.rstchanges/<PR#>.nsclean.rstchanges/<PR#>.imprint.rstchanges/<PR#>.background.rstchanges/<PR#>.extract_2d.rstchanges/<PR#>.master_background.rstchanges/<PR#>.wavecorr.rstchanges/<PR#>.srctype.rstchanges/<PR#>.straylight.rstchanges/<PR#>.wfss_contam.rstchanges/<PR#>.flatfield.rstchanges/<PR#>.fringe.rstchanges/<PR#>.pathloss.rstchanges/<PR#>.barshadow.rstchanges/<PR#>.photom.rstchanges/<PR#>.pixel_replace.rstchanges/<PR#>.resample_spec.rstchanges/<PR#>.residual_fringe.rstchanges/<PR#>.cube_build.rstchanges/<PR#>.extract_1d.rstchanges/<PR#>.resample.rststage 3
changes/<PR#>.assign_mtwcs.rstchanges/<PR#>.mrs_imatch.rstchanges/<PR#>.tweakreg.rstchanges/<PR#>.skymatch.rstchanges/<PR#>.exp_to_source.rstchanges/<PR#>.outlier_detection.rstchanges/<PR#>.tso_photometry.rstchanges/<PR#>.stack_refs.rstchanges/<PR#>.align_refs.rstchanges/<PR#>.klip.rstchanges/<PR#>.spectral_leak.rstchanges/<PR#>.source_catalog.rstchanges/<PR#>.combine_1d.rstchanges/<PR#>.ami.rstother
changes/<PR#>.wfs_combine.rstchanges/<PR#>.white_light.rstchanges/<PR#>.cube_skymatch.rstchanges/<PR#>.engdb_tools.rstchanges/<PR#>.guider_cds.rst