wip(iac): deploy new dev environment [pulumi up]

wip: separate build and develop jobs so that we can be able to re-use deploy on multiple environments.

WIP: realm, environment, backend

REALM, ENVIRONMENT:
Both are working but the following is still open for these two
besides various TODO in the code the iac/README.MD needs to be updated, the infra diagram from lucid added and also the mermaid diagram

BACKEND: preview works, but could not test if the rest works

wip: deploy test-realm realm.
add: documentation
remove: default gcp_api_gateway_config.yaml
wip: add deploy script

WIP:
manage the Identity Platform’s lifecycle with a dynamic resource
.env should use the fully qualified stack name
resource name should not have the realm or env name

wip: setup_env.py
wip: upload-artifacts.sh, deploy and destroy scripts.

wip: wait for update identity toolkit config to apply

separate upload-backend-artifacts.sh and upload-backend-artifacts.sh

wip: api gateway config mini refactor

wip: upload-backend-artifacts.sh, upload-frontend-artifacts.sh intermediate commit

wip: Github test/build pipeline is working (without deploy)

wip: github deploy to dev [pulumi up].

try: using backend: .env.example

clean up: build and upload scripts

clean up: remove Pulumi.<environment>.yaml and unused github workflows.

fix(backend): to separate target_environment_name and target_environment_type:

Target environment type: is used to know when to allow local development when setting CORS Policy.
Target environment name: required to know the environment some sentry events occurred.

wip: Github pipeline and configuration versioning

wip: upload templates

wip: setup_env script

fix: build and upload github pipeline

wip: set-up-env, prepare and up scripts

wip: refactor branch name formatters

fix: add a hash on the artifacts name to ensure uniqueness

fix: generate_esco_embeddings.py to have a possibility to generate indexes only.

wip: Generate indexes

wip:
Show the actual secret
run setup.py for each env pinning the release
- deploy dev and expect to cik the correct secrets files

feat(auth): use custom domains

chore(docs): Add necessary docs to deploy

wip: ssl_status

chore(docs): deployment-procedure.md

wip: identity platform setup with email template and custom domain support, add dns stack

wip(deploy-compass-realm): fix auth deploymnent.

Author: irumvanselme

.github/workflows/backend-ci.yml

@@ -0,0 +1,85 @@
+name: Backend CI & Artifact Upload
+
+on:
+  workflow_call:
+    inputs:
+      upload-artifacts:
+        required: true
+        type: boolean
+        description: 'Whether to upload deployable artifacts'
+
+jobs:
+  test-build-and-upload:
+    runs-on: ubuntu-latest
+    steps:
+      # setup.
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Setup python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Load cached Poetry installation
+        id: cached-poetry
+        uses: actions/cache@v4
+        with:
+          path: ~/.local
+          key: poetry-0
+
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          version: 1.8.5
+
+      - name: Load cached Poetry cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pypoetry
+          key: poetry-cache-${{ runner.os }}-${{ steps.setup_python.outputs.python-version }}-${{ env.POETRY_VERSION }}
+
+      - name: Install dependencies
+        shell: bash
+        run: |
+          poetry lock --no-update --no-interaction
+          poetry install --no-interaction
+        working-directory: backend
+
+      # test and lint
+
+      - name: Linting Bandit
+        shell: bash
+        run: poetry run bandit -c bandit.yaml -r .
+        working-directory: backend
+
+      - name: Linting Pylint
+        shell: bash
+        # Do not fail the build if linting errors (--exit-zero)
+        # Once we have fixed all the linting errors, we can remove this flag.
+        run: poetry run pylint --exit-zero --recursive=y .
+        working-directory: backend
+
+      - name: Copy the template .env.example to .env
+        run: cp backend/.env.example backend/.env
+
+      - name: Run unit tests
+        shell: bash
+        run: poetry run pytest -m 'not (evaluation_test or smoke_test)'
+        working-directory: backend
+
+      # build and upload artifacts
+
+      - name: Authenticate to google cloud
+        if: ${{ inputs.upload-artifacts }}
+        uses: google-github-actions/[email protected]
+        with:
+          credentials_json:
+            ${{ secrets.GCP_LOWER_ENVS_SERVICE_ACCOUNT_JSON }}
+
+      - name: Run build and upload script.
+        shell: bash
+        if: ${{ inputs.upload-artifacts }}
+        run: |
+          ./iac/scripts/build-and-upload-be.sh ${{ vars.ARTIFACT_REGISTRY_REGION }} ${{ secrets.GCP_REALM_ROOT_PROJECT_ID }} $GITHUB_STEP_SUMMARY $GITHUB_RUN_NUMBER

.github/workflows/build-frontend.yml

@@ -0,0 +1,23 @@
+name: Upload configurations and templates
+
+on:
+  workflow_call:
+
+jobs:
+  upload-templates:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Authenticate to google cloud
+        id: auth
+        uses: google-github-actions/[email protected]
+        with:
+          credentials_json:
+            ${{ secrets.GCP_LOWER_ENVS_SERVICE_ACCOUNT_JSON }}
+
+      - name: upload templates
+        working-directory: iac
+        run: |
+          ./scripts/upload-templates.sh ${{ vars.ARTIFACT_REGISTRY_REGION }} ${{ secrets.GCP_REALM_ROOT_PROJECT_ID }}