Run without docker (#649)

* Remove unused docker stacks

Update dev stacks to silence warnings

* Add start-bare-metal command to corgi cli

Effectively runs the same command as docker entrypoint

* Add build-frontend command; update fronted packages

Add python version and nvmrc

* Make environ optional for start-bare-metal

Easier to test this way

* Check python version matches docker version

* Add install script and source .env before start

* Add script to install secrets with aws ssm

For launch template

Add secrets spec; update install_secrets

For more information about how this secrets spec works, see openstax/aws-ruby repository

Turns out top_key is not optional in secrets spec

That README in openstax/aws-ruby should really be updated

When the secrets factory is being created, `.to_sym` is called on the top_key

That means it cannot be nil

Then top_key is used as a key for the secrets spec, even if it is an empty string

Consequently, top_key is required

* Support url-safe base64 encoding for session secret

* Add gunicorn logging configuration options

* Remove unused scripts

* Add .poetry-version file

We have had problems with poetry lock file compatibility in the past

This file will be read by the ansible playbook to determine which version of poetry to install

* Remove hotdog README section

This will need to work differently in the future if it's ever revived
Leaving the scripts for now as they might still be useful

* install_secrets: Ensure secret_name and secret_value have values for each secret

* Small update to corgi cli and ci tests

* Bump version for rollup-plugin-css-only

This version should be idempotent (hopefully)

* Get session secret from substitution

Author: TylerZeroMaster

.github/workflows/test.yml

@@ -56,15 +56,24 @@ jobs:
       - name: Frontend Unit Tests
         run: |
           set -e
-          docker compose -f docker-compose.stack.dev.yml exec frontend npm run test:unit
-          docker compose -f docker-compose.stack.dev.yml exec frontend npm run lint
+          ./corgi compose-do dev exec frontend npm run test:unit
+          ./corgi compose-do dev exec frontend npm run lint
 
       - name: Backend Unit Tests
         run: |
           set -e
-          docker compose -f docker-compose.stack.dev.yml exec backend pytest -vvv ./tests/unit
-          docker compose -f docker-compose.stack.dev.yml exec backend ruff check
-          docker compose -f docker-compose.stack.dev.yml exec backend ruff format --check
+          # We ignore patch version for this comparison
+          python_version_docker="$(./corgi compose-do dev exec backend python --version | cut -d' ' -f2 | cut -d. -f-2)"
+          python_version_repo="$(< .python-version)"
+          if [[ "$python_version_repo" != "$python_version_docker" ]]; then
+            echo "Python version mismatch:"
+            echo "Repo:   $python_version_repo"
+            echo "Docker: $python_version_docker"
+            exit 1
+          fi
+          ./corgi compose-do dev exec backend pytest -vvv ./tests/unit
+          ./corgi compose-do dev exec backend ruff check
+          ./corgi compose-do dev exec backend ruff format --check
 
       - name: Test database
         run: |
@@ -78,6 +87,6 @@ jobs:
           fi
           # Ensure the latest migration works backwards and forwards with 
           # existing job data
-          docker compose -f docker-compose.stack.dev.yml exec backend alembic downgrade -1
-          docker compose -f docker-compose.stack.dev.yml exec backend alembic upgrade head
+          ./corgi compose-do dev exec backend alembic downgrade -1
+          ./corgi compose-do dev exec backend alembic upgrade head
           ./corgi stop

.poetry-version

@@ -0,0 +1 @@
+1.8.3

.python-version

@@ -0,0 +1 @@
+3.11

README.md

@@ -307,43 +307,6 @@ with requests.session() as session:
     print(jobs.json())
 ```
 
-## Testing Unmerged CORGI & Enki Changes in Concourse
-
-[CORGI Hotdog](https://corgi-hotdog.ce.openstax.org/) is a testing environment for experimenting with changes before they go to staging.
-
-URL: https://corgi-hotdog.ce.openstax.org/
-
-### Use cases for hotdog
-- Catching issues that do not appear on Enki GitHub actions but does error in Concourse
-- Experimenting with changes that would be difficult/tedious to test locally (changes to concourse resource, upload steps, etc.)
-
-### Deploy Porcess
-
-- Two ways to deploy changes
-    - https://corgi-hotdog.ce.openstax.org/hotdog
-        * Enter the ref for one or both repos into the fields (ref can be commit sha or branch name)
-    - There is a script in [ce-scripts](https://github.com/openstax/ce-scripts/blob/c6c2e63d8941392003a4462c8a73e86aa06a598e/bash/hotdog)
-        * Run this script in Enki or CORGI.
-        * You can either specify a ref as an argument to the script or use your current ref by omitting this argument.
-- At this point, the two ways converge and the following occurs
-    - Success message
-    - On concourse, corgi-hotdog, wait for hotdog-head to get new checked out version, this triggers a concourse build pipeline is set in build-deploy-Enki
-        - See api call: `corgi-hotdog.openstax.org/hotdog/head` if you would like more details
-    - Since concourse depends on production Enki tags on dockerhub, there isn’t a way to do concourse + dev Enki tag — except thru hotdog
-    - Hotdog tag on dockerhub: corgi-hotdog rebuild as whatever dev ref you give it, then rebuilds the docker image for Enki.
-        - Rebuild triggered by submitting a new ref.
-    - Wait to build hotdog tag and push to dockerhub. At this point, you can create a job. still have to wait for concourse (steps: corgi-git-pdf & corgi-resource) to fetch the tag. then eventually the job will run.
-- Some additional details are
-    - If you checkout a branch, you will need to checkout the branch again if you want to pull the latest changes from the branch
-    - Jobs run on concourse. See logging & progress details on concourse/CORGI.
-    - In hotdog corgi ui: worker-version tells you what Enki ref it is & a timestamp
-    - Only one corgi-hotdog tag at a time
-
-### Hotdog TODO
-1. Automatically pull changes from branches
-1. Automatically redeploy the stack if python dependencies change (maybe add a field to corgi refs that, when set, will trigger deploy on checkout?)
-1. Consider creating a hybrid of PR pipeline so each PR can have a hotdog stack
-
 ## Generating an ERD
 
 The corgi CLI supports generating an ERD from database data. The auto-generated ERD, as well as a small description, replaces the README section labeled `## CORGI ERD`.

backend/app/app/core/auth.py

@@ -1,4 +1,5 @@
-from base64 import b64decode, b64encode
+import binascii
+from base64 import b64decode, b64encode, urlsafe_b64decode, urlsafe_b64encode
 from datetime import datetime, timedelta, timezone
 from typing import List, Optional, cast
 
@@ -15,10 +16,21 @@
 COOKIE_NAME = "user"
 
 
+def new_fernet_key(secret: str | None):
+    assert secret is not None, "Expected base64 encoded secret, got None"
+    try:
+        decoded = b64decode(secret)
+    except binascii.Error:
+        decoded = urlsafe_b64decode(secret)
+    assert len(decoded) >= 32, "Secret should be at least 32 bytes long"
+    # Fernet key must be 32 url-safe base64-encoded bytes.
+    return urlsafe_b64encode(decoded[:32])
+
+
 class Crypto:
     """Simple delegate class to simplify to encrypting/decrypting strings"""
 
-    f = Fernet(b64encode(b64decode(cast(str, SESSION_SECRET))[:32]))
+    f = Fernet(new_fernet_key(SESSION_SECRET))
 
     @staticmethod
     def encrypt(msg: str, encoding: str = "utf-8") -> str:

backend/app/bin/live-reload.sh

@@ -7,4 +7,8 @@ CALLABLE_NAME=${CALLABLE_NAME:-server}
 SESSION_SECRET="$(dd if=/dev/urandom bs=1024 count=1 2>/dev/null | base64)"
 export SESSION_SECRET
 
-uvicorn "${MODULE_NAME}:${CALLABLE_NAME}" --host 0.0.0.0 --port 80 --debug
+if [[ -z "${GUNICORN_CONF:-}" ]]; then
+    exec uvicorn "${MODULE_NAME}:${CALLABLE_NAME}" --host 0.0.0.0 --port 80 --debug
+else
+    exec gunicorn -k uvicorn.workers.UvicornWorker -c "$GUNICORN_CONF" "${MODULE_NAME}:${CALLABLE_NAME}"
+fi

backend/backend.dockerfile

@@ -53,7 +53,7 @@ ENV PYTHONPATH="/app:$PYTHONPATH"
 COPY ./docker/start.sh /start.sh
 RUN chmod +x /start.sh
 
-COPY ./docker/gunicorn.conf /gunicorn.conf
+COPY ./docker/gunicorn.conf.py /gunicorn.conf.py
 
 COPY ./app /app
 WORKDIR /app
@@ -83,7 +83,7 @@ ENV PYTHONPATH="/app:$PYTHONPATH"
 COPY ./docker/start.sh /start.sh
 RUN chmod +x /start.sh
 
-COPY ./docker/gunicorn.conf /gunicorn.conf
+COPY ./docker/gunicorn.conf.py /gunicorn.conf.py
 
 COPY ./app /app
 WORKDIR /app

backend/docker/gunicorn.conf backend/docker/gunicorn.conf.pybackend/docker/gunicorn.conf renamed to backend/docker/gunicorn.conf.py

@@ -8,10 +8,10 @@
 port = os.getenv("PORT", "80")
 bind_env = os.getenv("BIND", None)
 use_loglevel = os.getenv("LOG_LEVEL", "info")
-if bind_env:
-    use_bind = bind_env
-else:
-    use_bind = f"{host}:{port}"
+error_log_file = os.getenv("ERROR_LOG_FILE") or "-"
+access_log_file = os.getenv("ACCESS_LOG_FILE") or None
+syslog_conf = os.getenv("SYSLOG_ADDR") or None
+use_bind = bind_env if bind_env else f"{host}:{port}"
 
 cores = multiprocessing.cpu_count()
 workers_per_core = float(workers_per_core_str)
@@ -22,12 +22,22 @@
 else:
     web_concurrency = max(int(default_web_concurrency), 2)
 
+if syslog_conf and not (
+    syslog_conf.startswith("unix://")
+    or syslog_conf.startswith("udp://")
+    or syslog_conf.startswith("tcp://")
+):
+    raise ValueError(f"Invalid syslog address: {syslog_conf}")
+
 # Gunicorn config variables
 loglevel = use_loglevel
 workers = web_concurrency
 bind = use_bind
 keepalive = 120
-errorlog = "-"
+errorlog = error_log_file
+accesslog = access_log_file
+syslog = syslog_conf is not None
+syslog_addr = syslog_conf
 timeout = 120
 
 # For debugging and testing

backend/docker/start.sh

@@ -4,7 +4,7 @@ set -e
 
 MODULE_NAME=${MODULE_NAME:-app.main}
 CALLABLE_NAME=${CALLABLE_NAME:-server}
-GUNICORN_CONF=${GUNICORN_CONF:-/gunicorn.conf}
+GUNICORN_CONF=${GUNICORN_CONF:-/gunicorn.conf.py}
 PRE_START_PATH=${PRE_START_PATH:-/app/bin/prestart.sh}
 
 export APP_MODULE=${APP_MODULE:-"$MODULE_NAME:$CALLABLE_NAME"}