Skip to content

Commit

Permalink
Merge branch 'main' into flux_drop
Browse files Browse the repository at this point in the history
  • Loading branch information
tapastro authored Dec 6, 2024
2 parents 00669a7 + f295fbd commit cbb8d2a
Show file tree
Hide file tree
Showing 639 changed files with 26,316 additions and 15,896 deletions.
4 changes: 2 additions & 2 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ updates:
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
target-branch: "master"
target-branch: "main"
schedule:
interval: "weekly"
reviewers:
Expand All @@ -13,7 +13,7 @@ updates:
# Maintain dependencies for pip
- package-ecosystem: "pip"
directory: "/"
target-branch: "master"
target-branch: "main"
schedule:
interval: "weekly"
reviewers:
Expand Down
102 changes: 93 additions & 9 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,96 @@ Closes #
<!-- describe the changes comprising this PR here -->
This PR addresses ...

**Checklist for maintainers**
- [ ] added entry in `CHANGES.rst` within the relevant release section
- [ ] updated or added relevant tests
- [ ] updated relevant documentation
- [ ] added relevant milestone
- [ ] added relevant label(s)
- [ ] ran regression tests, post a link to the Jenkins job below.
[How to run regression tests on a PR](https://github.com/spacetelescope/jwst/wiki/Running-Regression-Tests-Against-PR-Branches)
- [ ] Make sure the JIRA ticket is [resolved properly](https://github.com/spacetelescope/jwst/wiki/How-to-resolve-JIRA-issues)
<!-- if you can't perform these tasks due to permissions, please ask a maintainer to do them -->
## Tasks
- [ ] **request a review from someone specific**, to avoid making the maintainers review every PR
- [ ] add a build milestone, i.e. `Build 11.3` (use the [latest build](https://github.com/spacetelescope/jwst/milestones) if not sure)
- [ ] Does this PR change user-facing code / API? (if not, label with `no-changelog-entry-needed`)
- [ ] write news fragment(s) in `changes/`: `echo "changed something" > changes/<PR#>.<changetype>.rst` (see below for change types)
- [ ] update or add relevant tests
- [ ] update relevant docstrings and / or `docs/` page
- [ ] [start a regression test](https://github.com/spacetelescope/RegressionTests/actions/workflows/jwst.yml) and include a link to the running job ([click here for instructions](https://github.com/spacetelescope/RegressionTests/blob/main/docs/running_regression_tests.md))
- [ ] Do truth files need to be updated ("okified")?
- [ ] **after the reviewer has approved these changes**, run `okify_regtests` to update the truth files
- [ ] if a JIRA ticket exists, [make sure it is resolved properly](https://github.com/spacetelescope/jwst/wiki/How-to-resolve-JIRA-issues)

<details><summary>news fragment change types...</summary>

- ``changes/<PR#>.general.rst``: infrastructure or miscellaneous change
- ``changes/<PR#>.docs.rst``
- ``changes/<PR#>.stpipe.rst``
- ``changes/<PR#>.datamodels.rst``
- ``changes/<PR#>.scripts.rst``
- ``changes/<PR#>.fits_generator.rst``
- ``changes/<PR#>.set_telescope_pointing.rst``
- ``changes/<PR#>.pipeline.rst``

## stage 1
- ``changes/<PR#>.group_scale.rst``
- ``changes/<PR#>.dq_init.rst``
- ``changes/<PR#>.emicorr.rst``
- ``changes/<PR#>.saturation.rst``
- ``changes/<PR#>.ipc.rst``
- ``changes/<PR#>.firstframe.rst``
- ``changes/<PR#>.lastframe.rst``
- ``changes/<PR#>.reset.rst``
- ``changes/<PR#>.superbias.rst``
- ``changes/<PR#>.refpix.rst``
- ``changes/<PR#>.linearity.rst``
- ``changes/<PR#>.rscd.rst``
- ``changes/<PR#>.persistence.rst``
- ``changes/<PR#>.dark_current.rst``
- ``changes/<PR#>.charge_migration.rst``
- ``changes/<PR#>.jump.rst``
- ``changes/<PR#>.clean_flicker_noise.rst``
- ``changes/<PR#>.ramp_fitting.rst``
- ``changes/<PR#>.gain_scale.rst``

## stage 2
- ``changes/<PR#>.assign_wcs.rst``
- ``changes/<PR#>.badpix_selfcal.rst``
- ``changes/<PR#>.msaflagopen.rst``
- ``changes/<PR#>.nsclean.rst``
- ``changes/<PR#>.imprint.rst``
- ``changes/<PR#>.background.rst``
- ``changes/<PR#>.extract_2d.rst``
- ``changes/<PR#>.master_background.rst``
- ``changes/<PR#>.wavecorr.rst``
- ``changes/<PR#>.srctype.rst``
- ``changes/<PR#>.straylight.rst``
- ``changes/<PR#>.wfss_contam.rst``
- ``changes/<PR#>.flatfield.rst``
- ``changes/<PR#>.fringe.rst``
- ``changes/<PR#>.pathloss.rst``
- ``changes/<PR#>.barshadow.rst``
- ``changes/<PR#>.photom.rst``
- ``changes/<PR#>.pixel_replace.rst``
- ``changes/<PR#>.resample_spec.rst``
- ``changes/<PR#>.residual_fringe.rst``
- ``changes/<PR#>.cube_build.rst``
- ``changes/<PR#>.extract_1d.rst``
- ``changes/<PR#>.resample.rst``

## stage 3
- ``changes/<PR#>.assign_mtwcs.rst``
- ``changes/<PR#>.mrs_imatch.rst``
- ``changes/<PR#>.tweakreg.rst``
- ``changes/<PR#>.skymatch.rst``
- ``changes/<PR#>.exp_to_source.rst``
- ``changes/<PR#>.outlier_detection.rst``
- ``changes/<PR#>.tso_photometry.rst``
- ``changes/<PR#>.stack_refs.rst``
- ``changes/<PR#>.align_refs.rst``
- ``changes/<PR#>.klip.rst``
- ``changes/<PR#>.spectral_leak.rst``
- ``changes/<PR#>.source_catalog.rst``
- ``changes/<PR#>.combine_1d.rst``
- ``changes/<PR#>.ami.rst``

## other
- ``changes/<PR#>.wfs_combine.rst``
- ``changes/<PR#>.white_light.rst``
- ``changes/<PR#>.cube_skymatch.rst``
- ``changes/<PR#>.engdb_tools.rst``
- ``changes/<PR#>.guider_cds.rst``
</details>
7 changes: 4 additions & 3 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,21 +16,22 @@ jobs:
name: check that `requirements-sdp.txt` is populated
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- run: grep -v '^ *#' requirements-sdp.txt
build:
needs: [ check ]
if: always() && (needs.check.result == 'success' || needs.check.result == 'skipped')
uses: OpenAstronomy/github-actions-workflows/.github/workflows/publish.yml@v1
uses: OpenAstronomy/github-actions-workflows/.github/workflows/publish.yml@d68193b68216da64eafaa618f53c59f5d271c56e # v1.14.0
with:
upload_to_pypi: ${{ (github.event_name == 'release') && (github.event.action == 'released') }}
targets: |
# Linux wheels
- cp3*-manylinux_x86_64
# MacOS wheels
- cp3*-macosx_x86_64
# Until we have arm64 runners, we can't automatically test arm64 wheels
# MacOS arm64 wheels
- cp3*-macosx_arm64
sdist: true
test_command: python -c "from jwst.lib import winclip; from jwst.cube_build import cube_match_internal, cube_match_sky_pointcloud, cube_match_sky_driz, blot_median; from jwst.straylight import calc_xart"
secrets:
pypi_token: ${{ secrets.PYPI_PASSWORD_STSCI_MAINTAINER }}
40 changes: 32 additions & 8 deletions .github/workflows/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,40 @@
name: Ensure changelog
name: changelog

on:
pull_request:
types: [labeled, unlabeled, opened, synchronize, reopened]
types:
- labeled
- unlabeled
- opened
- synchronize
- reopened

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
ensure_changelog:
name: Verify that a changelog entry exists for this pull request
check:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-changelog-entry-needed') }}
runs-on: ubuntu-latest
steps:
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: 3
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
- run: pip install .
- run: pip install towncrier
- run: towncrier check
- run: towncrier build --draft | grep -P '#${{ github.event.number }}'
prevent_manually_editing_changlog:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'allow-manual-changelog-edit') }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
submodules: true
- run: grep -P '\[[^\]]*#${{github.event.number}}[,\]]' CHANGES.rst
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-changelog-entry-needed') }}
fetch-depth: 0
- name: prevent direct changes to `CHANGES.rst`
run: git diff HEAD ${{ github.event.pull_request.base.sha }} --no-patch --exit-code CHANGES.rst

52 changes: 40 additions & 12 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,43 +3,71 @@ name: test
on:
push:
branches:
- master
- main
- '*x'
tags:
- '*'
pull_request:
branches:
- master
- main
schedule:
# Weekly Monday 9AM build
- cron: "0 9 * * 1"
workflow_dispatch:
inputs:
crds_context:
description: CRDS context (leave blank for latest)
type: string
required: false
default: ''
crds_server:
description: CRDS server
type: string
required: false
default: https://jwst-crds.stsci.edu

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.crds_context }}
cancel-in-progress: true

jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: '3.12'
- uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1
check:
uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v1
uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@d68193b68216da64eafaa618f53c59f5d271c56e # v1.14.0
with:
default_python: "3.12"
envs: |
- linux: check-style
- linux: check-security
- linux: check-dependencies
crds_contexts:
uses: spacetelescope/crds/.github/workflows/contexts.yml@master
- linux: check-types
latest_crds_contexts:
uses: ./.github/workflows/contexts.yml
crds_context:
needs: [ latest_crds_contexts ]
runs-on: ubuntu-latest
steps:
- id: context
run: echo context=${{ github.event_name == 'workflow_dispatch' && (inputs.crds_context != '' && inputs.crds_context || needs.latest_crds_contexts.outputs.jwst) || needs.latest_crds_contexts.outputs.jwst }} >> $GITHUB_OUTPUT
outputs:
context: ${{ steps.context.outputs.context }}
test:
uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v1
needs: [ crds_contexts ]
uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@d68193b68216da64eafaa618f53c59f5d271c56e # v1.14.0
needs: [ crds_context ]
with:
setenv: |
CRDS_PATH: /tmp/data/crds_cache
CRDS_SERVER_URL: https://jwst-crds.stsci.edu
CRDS_SERVER_URL: ${{ github.event_name == 'workflow_dispatch' && inputs.crds_server || 'https://jwst-crds.stsci.edu' }}
CRDS_CONTEXT: ${{ needs.crds_context.outputs.context }}
CRDS_CLIENT_RETRY_COUNT: 3
CRDS_CLIENT_RETRY_DELAY_SECONDS: 20
cache-path: /tmp/data/crds_cache
cache-key: crds-${{ needs.crds_contexts.outputs.jwst }}
cache-key: crds-${{ needs.crds_context.outputs.context }}
envs: |
- linux: py310-oldestdeps-xdist-cov
pytest-results-summary: true
Expand Down
36 changes: 30 additions & 6 deletions .github/workflows/ci_cron.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,46 @@ on:
# Weekly Monday 6AM build
- cron: "0 0 * * 1"
workflow_dispatch:
inputs:
crds_context:
description: CRDS context (leave blank for latest)
type: string
required: false
default: ''
crds_server:
description: CRDS server
type: string
required: false
default: https://jwst-crds.stsci.edu

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.crds_context }}
cancel-in-progress: true

jobs:
crds_contexts:
uses: spacetelescope/crds/.github/workflows/contexts.yml@master
latest_crds_contexts:
uses: ./.github/workflows/contexts.yml
crds_context:
needs: [ latest_crds_contexts ]
runs-on: ubuntu-latest
steps:
- id: context
run: echo context=${{ github.event_name == 'workflow_dispatch' && (inputs.crds_context != '' && inputs.crds_context || needs.latest_crds_contexts.outputs.jwst) || needs.latest_crds_contexts.outputs.jwst }} >> $GITHUB_OUTPUT
outputs:
context: ${{ steps.context.outputs.context }}
test:
if: (github.repository == 'spacetelescope/jwst' && (github.event_name == 'schedule' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run scheduled tests')))
uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v1
needs: [ crds_contexts ]
uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@d68193b68216da64eafaa618f53c59f5d271c56e # v1.14.0
needs: [ crds_context ]
with:
setenv: |
CRDS_PATH: /tmp/crds_cache
CRDS_SERVER_URL: https://jwst-crds.stsci.edu
CRDS_SERVER_URL: ${{ github.event_name == 'workflow_dispatch' && inputs.crds_server || 'https://jwst-crds.stsci.edu' }}
CRDS_CONTEXT: ${{ needs.crds_context.outputs.context }}
CRDS_CLIENT_RETRY_COUNT: 3
CRDS_CLIENT_RETRY_DELAY_SECONDS: 20
cache-path: /tmp/crds_cache
cache-key: crds-${{ needs.crds_context.outputs.jwst }}
cache-key: crds-${{ needs.crds_context.outputs.context }}
envs: |
- macos: py310-xdist
- macos: py312-xdist
Expand Down
26 changes: 26 additions & 0 deletions .github/workflows/contexts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: contexts

on:
workflow_call:
outputs:
jwst:
value: ${{ jobs.contexts.outputs.jwst }}
workflow_dispatch:

jobs:
contexts:
name: retrieve latest CRDS contexts
runs-on: ubuntu-latest
outputs:
jwst: ${{ steps.jwst_crds_context.outputs.pmap }}
steps:
- id: jwst_crds_context
env:
OBSERVATORY: jwst
CRDS_SERVER_URL: https://jwst-crds.stsci.edu
run: >
echo "pmap=$(
curl -s -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["${{ env.OBSERVATORY }}", null], "id": 1}' ${{ env.CRDS_SERVER_URL }}/json/ --retry 8 --connect-timeout 10 |
python -c "import sys, json; print(json.load(sys.stdin)['result'])"
)" >> $GITHUB_OUTPUT
- run: if [[ ! -z "${{ steps.jwst_crds_context.outputs.pmap }}" ]]; then echo ${{ steps.jwst_crds_context.outputs.pmap }}; else exit 1; fi
2 changes: 1 addition & 1 deletion .github/workflows/label_pull_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
triage:
runs-on: ubuntu-latest
steps:
- uses: actions/labeler@v5
- uses: actions/labeler@8558fd74291d67161a8a78ce36a881fa63b766a9 # v5.0.0
if: github.event_name == 'pull_request_target' || github.event_name == 'pull_request'
with:
repo-token: "${{ secrets.GITHUB_TOKEN }}"
Loading

0 comments on commit cbb8d2a

Please sign in to comment.