diff --git a/.env b/.env index e77a955..c1c4223 100644 --- a/.env +++ b/.env @@ -1,7 +1,11 @@ PYTHON_IMPLEMENTATION=python -PYTHON_VERSION=3.10 +PYTHON_VERSION=3.12 MARIADB_VERSION=10.5 MYSQL_VERSION=8.0 ORACLE_VERSION=23.5.0.0 POSTGRESQL_VERSION=14 POSTGIS_VERSION=3.1 +SQLITE_VERSION= +SQLITE_CFLAGS="-DSQLITE_ENABLE_DESERIALIZE \ + -DSQLITE_ENABLE_JSON1 \ + -DSQLITE_MAX_VARIABLE_NUMBER=32766" diff --git a/Containerfile b/Containerfile index c75ffeb..8c5be34 100644 --- a/Containerfile +++ b/Containerfile @@ -1,7 +1,7 @@ # syntax=docker/dockerfile:1.12 ARG PYTHON_IMPLEMENTATION=python -ARG PYTHON_VERSION=3.10 +ARG PYTHON_VERSION=3.12 FROM ${PYTHON_IMPLEMENTATION}:${PYTHON_VERSION}-slim-bookworm LABEL org.opencontainers.image.authors="Django Software Foundation" diff --git a/README.md b/README.md index a64b14e..dba7dfe 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Tooling and test execution support for [Django][0] :unicorn: 3. Build the image: ```console - $ docker compose build sqlite + $ docker compose build base ``` 4. Run the tests: @@ -193,7 +193,7 @@ environment variables: | ----------------------- | ------------- | ---------------------------------------------------- | | `DJANGO_PATH` | `../django` | Path to the Django repostory on your local machine | | `PYTHON_IMPLEMENTATION` | `python` | Implementation of Python to use — `python` or `pypy` | -| `PYTHON_VERSION` | `3.10` | Version of Python container image to use | +| `PYTHON_VERSION` | `3.12` | Version of Python container image to use | The versions of various backend services can be switched by setting these environment variables: @@ -204,7 +204,13 @@ The versions of various backend services can be switched by setting these enviro | `ORACLE_VERSION` | `23.5.0.0` | Version of Oracle container image to use | | `POSTGRESQL_VERSION` | `14` | Version of PostgreSQL container image to use | | `POSTGIS_VERSION` | `3.1` | Version of PostGIS extension to use | +| `SQLITE_VERSION` | | Version of SQLite to compile and use | +> [!NOTE] +> +> If left unspecified, the SQLite version provided by Debian will be used. +> Using a specific SQLite version requires compiling it from source. For more +> details, see [SQLite Versions](#SQLite-Versions). ### Python Versions @@ -212,7 +218,7 @@ The `PYTHON_VERSION` environment variable controls which version of Python you are running the tests against, e.g. ```console -$ PYTHON_VERSION=3.10 docker compose run --rm sqlite +$ PYTHON_VERSION=3.12 docker compose run --rm sqlite ``` In addition, it's possible to select a different implementation of Python, i.e. @@ -229,7 +235,8 @@ restrictions with respect to the range of versions available. ### Database Versions Most database container images are pulled from [Docker Hub][2]. Oracle database -is pulled from the [Oracle Container Registry][3]. +is pulled from the [Oracle Container Registry][3]. Specific versions of SQLite +are compiled directly from the tags in the [official Git mirror][11]. You can switch the version of the database you test against by changing the appropriate environment variable. Available options and their defaults can be @@ -273,6 +280,46 @@ To determine what database versions can be used you can check the release notes for the branch of Django that you have checked out, or alternatively there is the [supported database versions][4] page on Django's Trac Wiki. +#### SQLite Versions + +SQLite is normally bundled in the Python installation using the version +available on the system where Python is compiled. We use the Python Docker image +based on Debian `bookworm`, which has SQLite 3.40.1. + +To use a different version, we compile SQLite from source and load the library +dynamically using `LD_PRELOAD`. There are a few caveats as a result: + +- Some SQLite features are only available if certain flags are set during + compilation. SQLite is known to change these flags in newer releases, such as + to enable features by default that were previously opt-in. When Python is + compiled, it inspects the system's SQLite to determine features that are + included in the `sqlite` module. A mismatch in the module and the dynamically + loaded library may result in Python failing to load, which may happen if we + use an SQLite version that is older than the system version. +- Debian and Ubuntu use a custom `CFLAGS` variable to compile their distributed + SQLite. Historically, Django's CI has only been configured with SQLite + versions that come with the operating system. If SQLite is compiled with + different flags, some tests may fail. + +We currently work around the above caveats by setting the simplest `CFLAGS` +value that allows all the tests to pass. To customize the `CFLAGS` used for the +compilation, you can set the `SQLITE_CFLAGS` environment variable. See the +[`.env`][10] file for its default value. + +``` +SQLITE_VERSION=3.48.0 SQLITE_CFLAGS="-DSQLITE_OMIT_JSON -DSQLITE_MAX_VARIABLE_NUMBER=999" docker compose run --build --rm sqlite +``` + +> [!NOTE] +> +> The `--build` argument is necessary if you've changed `SQLITE_CFLAGS` since +> the last run, as it's not part of the image tag. You can also rebuild the +> image separately by running `docker compose build sqlite`, optionally with +> `--no-cache` to ignore the cached build. + +In the future, the Django codebase may be more robust when tested against +different SQLite configurations and the `CFLAGS` workaround may no longer be +necessary. ### Other Versions @@ -309,3 +356,5 @@ with no promises that they'll be delivered: [7]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-unit-tests [8]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-selenium-tests [9]: https://docs.djangoproject.com/en/stable/ref/contrib/gis/testing/#geodjango-tests +[10]: .env +[11]: https://github.com/sqlite/sqlite diff --git a/compose.yml b/compose.yml index fc65ac7..301fb3e 100644 --- a/compose.yml +++ b/compose.yml @@ -8,7 +8,7 @@ x-base: &base args: - PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION} - PYTHON_VERSION=${PYTHON_VERSION} - additional_contexts: + additional_contexts: &additional-contexts src: ${DJANGO_PATH:-../django} volumes: - ${DJANGO_PATH:-../django}:/django/source:rw @@ -139,6 +139,10 @@ volumes: services: + # Base service to allow building the image with `docker compose build base`. + base: + <<: *base + # Services: Databases mariadb-db: @@ -285,6 +289,38 @@ services: sqlite: <<: *base + image: django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}-sqlite${SQLITE_VERSION} + pull_policy: never + build: + context: . + dockerfile_inline: | + FROM django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION} + SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-o", "xtrace", "-c"] + # Only compile SQLite and set LD_PRELOAD if a version is specified. + RUN <