Move outlier detection utility functions from jwst to stcal #270

braingram · 2024-07-08T19:04:22Z

This PR moves a number of utility functions related to outlier detection from jwst to stcal.

This PR is required for spacetelescope/jwst#8613

As this PR adds a few new dependencies (and these are added to the devdeps) the run devdeps tests label was added. It's expected that the jwst devdeps will fail as they are failing on main: https://github.com/spacetelescope/stcal/actions/runs/9836681142/job/27152917937

Checklist

added entry in CHANGES.rst (either in Bug Fixes or Changes to API)
updated relevant tests
updated relevant documentation
updated relevant milestone(s)
added relevant label(s)

codecov · 2024-07-08T19:10:16Z

Codecov Report

Attention: Patch coverage is 96.73913% with 6 lines in your changes missing coverage. Please review.

Project coverage is 84.32%. Comparing base (58018d4) to head (2112773).
Report is 190 commits behind head on main.

Files with missing lines	Patch %	Lines
src/stcal/outlier_detection/utils.py	96.55%	3 Missing ⚠️
tests/outlier_detection/test_utils.py	96.90%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #270      +/-   ##
==========================================
+ Coverage   84.01%   84.32%   +0.31%     
==========================================
  Files          39       41       +2     
  Lines        7345     7529     +184     
==========================================
+ Hits         6171     6349     +178     
- Misses       1174     1180       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

braingram · 2024-07-10T14:27:05Z

Closing and opening to re-trigger CI since the run devdeps tests label didn't seem to take.

emolter

Looking good, just a few small things.

src/stcal/outlier_detection/utils.py

emolter

All my comments have been addressed, looks good to me

src/stcal/outlier_detection/utils.py

mcara

LGTM

tox.ini

emolter · 2024-07-26T12:43:47Z

looks like mypy needs an override in pyproject.toml for drizzle

kmacdonald-stsci · 2024-07-26T12:49:26Z

src/stcal/outlier_detection/utils.py

+
+
+def _abs_deriv(array):
+    tmp = np.zeros(array.shape, dtype=np.float64)


Missing docstring.

I added short docstrings in 2112773

This function (and _absolute_subtract) should likely not be used since they appear to introduce erroneous cr flags due to treating off-edge pixels as 0. However, fixing this bug would result in changes to regression test results and I think makes more sense in a separate PR (where the new results can be verified). There is a JP ticket for this spacetelescope/jwst#8636 and candidate new code here https://github.com/spacetelescope/jwst/pull/8635/files

For the scope of this PR I added "Do not use this function" to the docstrings as moving this code to stcal did not introduce the bug.

kmacdonald-stsci · 2024-07-26T12:49:38Z

src/stcal/outlier_detection/utils.py

+
+
+def _absolute_subtract(array, tmp, out):
+    tmp = np.abs(array - tmp)


Missing docstring.

I added a brief docstring to this function (see #270 (comment) for more details).

I attempted to add more but given the number of lines of code and the array "switch-a-roo" that this function is performing I think reading the code is the best bet for trying to understand what's happening.

Also this function will be removed if the candidate code for fixing spacetelescope/jwst#8636 is acceptable.

Tests were unable to produce a situation where the filtered warning could be produced. The goal was to add a more specific filter (one that uses a message). If the warning can be reproduced the filter with a message can be re-added. It's possible this warning is no longer produced and so this commit removes the filter.

braingram force-pushed the outlier_detection branch from 5afae74 to 4028dea Compare July 8, 2024 19:05

github-actions bot added the installation label Jul 8, 2024

braingram force-pushed the outlier_detection branch from fd15316 to 4df2447 Compare July 8, 2024 19:21

github-actions bot added the testing label Jul 8, 2024

braingram force-pushed the outlier_detection branch 2 times, most recently from ad63436 to 4733fb9 Compare July 10, 2024 00:02

github-actions bot added the documentation Improvements or additions to documentation label Jul 10, 2024

braingram changed the title ~~WIP: Outlier detection~~ Move outlier detection utility functions from jwst to stcal Jul 10, 2024

braingram added the run devdeps tests label Jul 10, 2024

braingram mentioned this pull request Jul 10, 2024

Outlier detection refactoring spacetelescope/jwst#8613

Merged

8 tasks

braingram closed this Jul 10, 2024

braingram reopened this Jul 10, 2024

braingram marked this pull request as ready for review July 10, 2024 17:15

braingram requested a review from a team as a code owner July 10, 2024 17:15

emolter reviewed Jul 11, 2024

View reviewed changes

braingram force-pushed the outlier_detection branch from c8885c5 to 30355b0 Compare July 11, 2024 21:07

emolter approved these changes Jul 18, 2024

View reviewed changes

src/stcal/outlier_detection/utils.py Show resolved Hide resolved

src/stcal/outlier_detection/utils.py Outdated Show resolved Hide resolved

braingram requested review from nden and mcara July 18, 2024 14:55

This was referenced Jul 18, 2024

Consolidate and organize utility functions #271

Open

switch docs theme to sphinx_rtd, unpin sphinx #272

Merged

braingram force-pushed the outlier_detection branch from c573846 to 4564878 Compare July 26, 2024 11:57

mcara approved these changes Jul 26, 2024

View reviewed changes

emolter reviewed Jul 26, 2024

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

kmacdonald-stsci approved these changes Jul 26, 2024

View reviewed changes

braingram added 2 commits July 26, 2024 08:54

add some outlier detection utility functions

c284f74

add medfilt

ac93830

braingram added 16 commits July 26, 2024 08:55

update deps devdeps

741edb3

WIP: adding unit tests

546fd07

more tests and some renaming

17631e5

renaming some functions

d9007b6

rename file

a957d99

change back to old abs_deriv

d7936bf

more tests

a57d618

fix test

6080156

add changelog

252ee7c

add docs

63b84f0

update docstrings

fd738a8

docs fixes

fc59880

remove partial comment

f3d34c2

fix docstring

4ebbeec

remove resample_data boolean argument

61aeee8

braingram force-pushed the outlier_detection branch from 89cf708 to 61aeee8 Compare July 26, 2024 12:55

braingram added 3 commits July 26, 2024 08:58

add mypy exclusion for drizzle

573b5d9

copy docstring from jwst

aa64f1d

update docstrings

2112773

braingram merged commit 809de23 into spacetelescope:main Jul 30, 2024
28 of 32 checks passed

braingram deleted the outlier_detection branch July 30, 2024 16:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move outlier detection utility functions from jwst to stcal #270

Move outlier detection utility functions from jwst to stcal #270

braingram commented Jul 8, 2024 •

edited

Loading

codecov bot commented Jul 8, 2024 •

edited

Loading

braingram commented Jul 10, 2024

emolter left a comment

emolter left a comment

mcara left a comment

emolter commented Jul 26, 2024

kmacdonald-stsci Jul 26, 2024

braingram Jul 26, 2024

kmacdonald-stsci Jul 26, 2024

braingram Jul 26, 2024



		def _abs_deriv(array):
		tmp = np.zeros(array.shape, dtype=np.float64)



		def _absolute_subtract(array, tmp, out):
		tmp = np.abs(array - tmp)

Move outlier detection utility functions from jwst to stcal #270

Move outlier detection utility functions from jwst to stcal #270

Conversation

braingram commented Jul 8, 2024 • edited Loading

codecov bot commented Jul 8, 2024 • edited Loading

Codecov Report

braingram commented Jul 10, 2024

emolter left a comment

Choose a reason for hiding this comment

emolter left a comment

Choose a reason for hiding this comment

mcara left a comment

Choose a reason for hiding this comment

emolter commented Jul 26, 2024

kmacdonald-stsci Jul 26, 2024

Choose a reason for hiding this comment

braingram Jul 26, 2024

Choose a reason for hiding this comment

kmacdonald-stsci Jul 26, 2024

Choose a reason for hiding this comment

braingram Jul 26, 2024

Choose a reason for hiding this comment

braingram commented Jul 8, 2024 •

edited

Loading

codecov bot commented Jul 8, 2024 •

edited

Loading