CDK v1 to v2 (#19)

Configure GH actions for testing/deployment (#4)

Updated GitHub Actions to trigger testing and deployment
for appropriate actions:

- Run tests on code in "head" branch for all PRs to `develop` or `main`
  (on open, reopen, edit, or synch of a PR)
- Run tests on code in "target" branch for all PRs closed/merged to
  `develop` or `main`
- Deploy to `dev` env on PR close/merge to `develop`
- Deploy to `prod` env on tag/release on `main`

Use GitHub OIDC for AWS credentials (#5)

- Allow manual workflow trigger in GitHub
- Add `TOX_OPTS` env var to `Makefile` to allow override of `tox`
  options from command line when running `make`
- Restrict integration tests to develop, main branches because
  OIDC trust policy is restricted to those branches
- Add permissions boundary to integration test resources stack (#6)

Run integration tests on PR (#7)

- To avoid having to wait for PR approval and merging to `develop` or
  `main`, run integration tests when PR is opened or updated.
- Ensure integration test resources are destroyed even when the stack
  under test fails to deploy.
- Simplify managed policy construction
- Export secrets as env vars for int tests
- Rename variable named the same as a builtin
- Remove redundant removal policy

- Add forward notification stack (#13)

- Update versions in pre-commit config: The previous version of the isort
  pre-commit hook was causing pre-commit to exit with a `CalledProcessError`.
  Running `pre-commit autoupdate` to update the hook versions fixed the problem,
  allowing pre-commit to again run successfully.
- Fix code coverage "no data collected" warning
- Upgrade GitHub actions; cache Python deps
- Fix broken historical test
- Refactor "historical" files to avoid conflicts w/"forward" files
- Pinned major versions of dependencies: The `moto` library recently released
  version 5.x, containing breaking changes, so it caused unit tests to fail due to
  import errors. Given the need to pin `moto` to 4.x, I also pinned all other
  dependencies to avoid other such problems.

- Make tiler queue optional (#14)

If no value is supplied for the env var HLS_LPDAAC_TILER_QUEUE_ARN,
a dummy queue will be created and used.  This is necessary because there
is only such a queue in production, so other envs need a dummy queue.

- Remove instructions on manual deployment from EC2 instance

Migrate CDK v1 to v2 (#17)

- Add CDK v2 bootstrap template (with MCP-compatible
  adjustments from the default template)
- Add CDK bootstrap command to Makefile
- Migrate from aws-cdk v1 to v2 packages

Bump CDK and Node versions (#20)

.github/workflows/deploy.yml

@@ -0,0 +1,68 @@
+name: Deploy
+
+on:
+  workflow_dispatch:
+  workflow_call:
+    inputs:
+      environment:
+        required: true
+        type: string
+      PYTHON_VERSION:
+        required: true
+        type: string
+      TOX_MIN_VERSION:
+        required: true
+        type: string
+
+defaults:
+  run:
+    shell: bash
+
+# See https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#updating-your-github-actions-workflow
+permissions:
+  id-token: write # required for requesting the JWT
+  contents: read  # required for actions/checkout
+
+jobs:
+  deploy:
+    runs-on: ubuntu-22.04
+    environment: ${{ inputs.environment }}
+    steps:
+      - name: Checkout source
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "${{ inputs.PYTHON_VERSION }}"
+          cache: 'pip'
+          cache-dependency-path: setup.py
+      - name: Install dependencies
+        run: |
+          pip install "tox>=${{ inputs.TOX_MIN_VERSION }}"
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          aws-region: ${{ vars.AWS_DEFAULT_REGION }}
+          role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME_ARN }}
+          role-session-name: ${{ github.actor }}
+      - name: Convert secrets to environment variables
+        env:
+          SECRETS_JSON: ${{ toJson(secrets) }}
+        run: |
+          while read -rd $'' line; do
+            echo "$line" >> $GITHUB_ENV
+          done < <(
+            jq -r <<<"$SECRETS_JSON" 'to_entries|map("\(.key)=\(.value)\u0000")[]'
+          )
+      - name: Convert vars to environment variables
+        env:
+          VARS_JSON: ${{ toJson(vars) }}
+        run: |
+          while read -rd $'' line; do
+            echo "$line" >> $GITHUB_ENV
+          done < <(
+            jq -r <<<"$VARS_JSON" 'to_entries|map("\(.key)=\(.value)\u0000")[]'
+          )
+      - name: Deploy forward notification to ${{ inputs.environment }}
+        run: |
+          make deploy-forward

.github/workflows/main.yml

@@ -1,26 +1,167 @@
 # Install Python dependencies, run tests, and lint with a single version of Python.
 # See https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
 
-name: HLS LPDAAC Historical
+name: HLS LPDAAC Notifications
 
 on:
+  workflow_dispatch:
+  release:
+    types:
+      - published
   push:
-    branches: [ main ]
+    branches:
+      - main
+      - develop
+    tags-ignore:
+      - '*'
+    paths:
+      - '.github/workflows/*'
+      - 'cdk/**'
+      - 'src/**'
+      - 'cdk.json'
+      - 'Makefile'
+      - 'setup.py'
+      - 'tox.ini'
   pull_request:
-    branches: [ main ]
+    types:
+      - edited
+      - opened
+      - reopened
+      - synchronize
+    branches:
+      - main
+      - develop
+    paths:
+      - '.github/workflows/*'
+      - 'cdk/**'
+      - 'src/**'
+      - 'cdk.json'
+      - 'Makefile'
+      - 'setup.py'
+      - 'tox.ini'
+
+# See https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#updating-your-github-actions-workflow
+permissions:
+  id-token: write # required for requesting the JWT
+  contents: read  # required for actions/checkout
+
+defaults:
+  run:
+    shell: bash
 
 jobs:
-  build:
-    runs-on: ubuntu-latest
+  config:
+    # This is a hack to work around the lack of support for two other possiblities for
+    # avoiding duplication of configuration values:
+    #
+    # (1) YAML anchors (https://yaml.org/spec/1.1/current.html#id899912) and aliases
+    #     (https://yaml.org/spec/1.1/current.html#id902561)
+    # (2) Availability of `env` context within `jobs.<job-id>.with.<with-id>` (see
+    #     https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability)
+    #
+    # Alternative hack: https://github.com/actions/runner/issues/1182#issuecomment-1262870831
+    runs-on: ubuntu-22.04
+    outputs:
+      PYTHON_VERSION: "3.9"
+      TOX_MIN_VERSION: "3.18.0"  # `allowlist_externals` replaces `whitelist_externals`
+    steps:
+      - name: Configure shared values
+        run: ""  # Nothing to do, but at least one step is required
+
+  unit-tests:
+    runs-on: ubuntu-22.04
+    needs: config
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python
-      uses: actions/setup-python@v2
-      with:
-        python-version: "3.9"
-    - name: Install dependencies
-      run: |
-        pip install tox
-    - name: Run tests
-      run: |
-        tox -e py
+      - name: Checkout source
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "${{ needs.config.outputs.PYTHON_VERSION }}"
+          cache: 'pip'
+          cache-dependency-path: setup.py
+      - name: Install dependencies
+        run: |
+          pip install "tox>=${{ needs.config.outputs.TOX_MIN_VERSION }}"
+      - name: Run unit tests
+        run: |
+          make unit-tests
+
+  integration-tests:
+    runs-on: ubuntu-22.04
+    environment: dev-forward
+    needs: config
+    steps:
+      - name: Checkout source
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "${{ needs.config.outputs.PYTHON_VERSION }}"
+          cache: 'pip'
+          cache-dependency-path: setup.py
+      - name: Install dependencies
+        run: |
+          pip install "tox>=${{ needs.config.outputs.TOX_MIN_VERSION }}"
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          aws-region: ${{ vars.AWS_DEFAULT_REGION }}
+          role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME_ARN }}
+          role-session-name: ${{ github.actor }}
+      - name: Convert secrets to environment variables
+        env:
+          SECRETS_JSON: ${{ toJson(secrets) }}
+        run: |
+          while read -rd $'' line; do
+            echo "$line" >> $GITHUB_ENV
+          done < <(
+            jq -r <<<"$SECRETS_JSON" 'to_entries|map("\(.key)=\(.value)\u0000")[]'
+          )
+      - name: Convert vars to environment variables
+        env:
+          VARS_JSON: ${{ toJson(vars) }}
+        run: |
+          while read -rd $'' line; do
+            echo "$line" >> $GITHUB_ENV
+          done < <(
+            jq -r <<<"$VARS_JSON" 'to_entries|map("\(.key)=\(.value)\u0000")[]'
+          )
+      - name: Deploy forward notification integration test stack
+        run: |
+          make deploy-forward-it
+      - name: Run forward notification integration tests
+        run: |
+          make forward-integration-tests
+      - name: Destroy forward notification integration test stack
+        if: '!cancelled()'
+        run: |
+          make destroy-forward-it
+
+  deploy-dev:
+    # Deploy to Dev only on push (including merged PR) to `develop` branch
+    if: github.event_name == 'push' && github.event.ref == 'refs/heads/develop'
+    needs:
+      - config
+      - unit-tests
+      - integration-tests
+    uses: ./.github/workflows/deploy.yml
+    with:
+      environment: dev-forward
+      PYTHON_VERSION: "${{ needs.config.outputs.PYTHON_VERSION }}"
+      TOX_MIN_VERSION: "${{ needs.config.outputs.TOX_MIN_VERSION }}"
+    secrets: inherit
+
+  deploy-prod:
+    # Deploy to Prod only on publishing a release (tag) on `main` branch
+    if: github.event_name == 'release' && github.event.action == 'published'
+    needs:
+      - config
+      - unit-tests
+      - integration-tests
+    uses: ./.github/workflows/deploy.yml
+    with:
+      environment: prod-forward
+      PYTHON_VERSION: "${{ needs.config.outputs.PYTHON_VERSION }}"
+      TOX_MIN_VERSION: "${{ needs.config.outputs.TOX_MIN_VERSION }}"
+    secrets: inherit

.gitignore

@@ -1,9 +1,9 @@
 .envrc
+.python-version
 
 ### CDK ###
 *.swp
 package-lock.json
-cdk.context.json
 
 # CDK asset staging directory
 .cdk.staging

.pre-commit-config.yaml

@@ -1,7 +1,7 @@
 default_stages: [commit]
 repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v4.1.0
+  rev: v4.5.0
   hooks:
   - id: check-executables-have-shebangs
   - id: check-merge-conflict
@@ -12,14 +12,14 @@ repos:
   - id: end-of-file-fixer
   - id: trailing-whitespace
 - repo: https://github.com/psf/black
-  rev: 22.3.0
+  rev: 24.1.0
   hooks:
   - id: black
 - repo: https://github.com/PyCQA/isort
-  rev: 5.10.1
+  rev: 5.13.0
   hooks:
   - id: isort
 - repo: https://github.com/PyCQA/flake8
-  rev: 4.0.1
+  rev: 7.0.0
   hooks:
   - id: flake8

.vscode/settings.json

@@ -1,16 +1,10 @@
 {
-  "deepscan.enable": true,
-  "python.formatting.provider": "black",
-  "python.linting.mypyEnabled": true,
-  "python.linting.enabled": true,
-  "python.linting.mypyArgs": [
-    "--show-column-numbers",
-    "src",
-    "tests"
-  ],
   "python.analysis.typeCheckingMode": "basic",
   "python.analysis.extraPaths": [
     "./src",
     "./tests"
-  ]
+  ],
+  "yaml.schemas": {
+    "https://json.schemastore.org/github-workflow.json": "file://./.github/workflows/deploy.yml"
+  }
 }