Merge pull request #1059 from overhangio/redwood

Upgrade to Redwood

Author: regisb

CHANGELOG.md

@@ -20,6 +20,45 @@ instructions, because git commits are used to generate release notes:
 
 <!-- scriv-insert-here -->
 
+<a id='changelog-18.0.0'></a>
+## v18.0.0 (2024-06-19)
+
+- 💥[Feature] Upgrade to Redwood (by @dawoudsheraz)
+- [Bugfix] Wrap Django5 warning imports in try-except block to avoid failures in django3 that's still in use in edx-platform's master branch (by @mariajgrimaldi).
+- 💥[Feature] Pull translations via `atlas` during Docker build. This breaks the `openedx-i18n` custom locale Tutor feature in favor of [OEP-58](https://docs.openedx.org/en/latest/developers/concepts/oep58.html) in favor of <https://github.com/openedx/openedx-translations>. (by @omarithawi)
+- 💥[Feature] The `openedx-assets` command is replaced with `npm run` subcommands. This will slightly reduce the build time for edx-platform assets and comprehensive themes. It will also open up the door for more significant build time reductions in the future. Here is a migration guide, where each command is to be run in the `lms` or `cms` container:
+
+  **Before**                               | **After**
+  -----------------------------------------|-------------------------------------------------------------------------------------
+  `openedx-assets build --env=prod ARGS`   | `npm run build -- ARGS`
+  `openedx-assets build --env=dev ARGS`    | `npm run build-dev -- ARGS`
+  `openedx-assets common --env=prod ARGS`  | `npm run compile-sass -- --skip-themes ARGS`
+  `openedx-assets common --env=dev ARGS`   | `npm run compile-sass-dev -- --skip-themes ARGS`
+  `openedx-assets webpack --env=prod ARGS` | `npm run webpack -- ARGS`
+  `openedx-assets webpack --env=dev ARGS`  | `npm run webpack-dev -- ARGS`
+  `openedx-assets npm`                     | `npm run postinstall` (`npm clean-install` runs this automatically)
+  `openedx-assets xmodule`                 | (no longer necessary)
+  `openedx-assets collect ARGS`            | `./manage.py lms collectstatic --noinput ARGS && ./manage.py cms collectstatic ARGS`
+  `openedx-assets watch-themes ARGS`       | `npm run watch-themes -- ARGS`
+
+For more details, see the [deprecation notice for paver](https://github.com/openedx/edx-platform/issues/34467)
+and the [static assets reference](https://github.com/openedx/edx-platform/tree/open-release/redwood.master/docs/references/static-assets.rst)
+in edx-platform.
+
+- 💥[Feature] Update MongoDB to v7.0.7 (by @dawoudsheraz) MongoDB is upgraded from version 4.4 to 7.0. Since there have been major releases since 4.4, the upgrade will need to go through them before running Mongo 7. MongoDB would need to follow 4.4 --> 5.0 --> 6.0 --> 7.0 upgrade path to work correctly. The container will keep on restarting with featureCompatibility error if the upgrade path is not followed. To upgrade mongo, run the following command based in the appropriate environment:
+
+    tutor <dev|local|k8s> upgrade --from=quince
+
+For k8s only, the above command will not perform the upgrade automatically. Instead, the command will output a series of commands that would need to be run manually to carry out the upgrade.
+
+- [Improvement] Upgrade Nodejs from 16.14.0 to 18.20.1 in edx-platform. (by @regisb)
+- [Improvement] Auto-detect bind mounts of openedx-learning for edx-platform (by @bradenmacdonald)
+- [Feature] Upgrade Open edX image to use Python 3.11 (by @dawoudsheraz)
+- [Bugfix] Remove CORS_ALLOW_HEADERS setting from the LMS/Studio config template. This setting, which holds site-agnostic application logic, is now consistently set to a reasonable value upstream by LMS and CMS config. Using the upstream values fixes a bug where course import in Studio using the new Course Authoring MFE was broken in Tutor deployments because it required additional headers to be allowed (content-range and content-disposition) (by @ormsbee)
+- [Improvement] Made Docker cache hits more frequent during the openedx image build via BuildKit's `COPY --link` feature (by @kdmccormick).
+- 💥[Improvement] Upgrade MySQL to 8.4.0. The upgrade should be automatic for most users. However, if you are running a third-party MySQL (i.e., RUN_MYSQL=false), you are expected to upgrade manually. Please refer to the third-party provider's documentation for detailed upgrade instructions. Ensuring that your MySQL version is up-to-date is crucial for maintaining compatibility and security. (by @rohansaeed)
+- 💥[Improvement] Ensure that the edx-platform repository git checkout is cached by Docker during image build. This means that the cache will automatically be cleared any time there is an upstream change. Thus, it is no longer necessary to run `tutor images build --no-cache` just to fetch the latest edx-platform changes. For this to work, any GitHub repository referenced by `EDX_PLATFORM_REPOSITORY` needs to end with ".git". Make sure that this is the case if you have modified the value of this setting in the past. (by @regisb)
+
 <a id='changelog-17.0.6'></a>
 ## v17.0.6 (2024-06-13)
 

docs/__init__.py

@@ -75,13 +75,13 @@ This configuration parameter defines which Caddy Docker image to use.
 
 This configuration parameter defines which Elasticsearch Docker image to use.
 
-- ``DOCKER_IMAGE_MONGODB`` (default: ``"docker.io/mongo:4.4.22"``)
+- ``DOCKER_IMAGE_MONGODB`` (default: ``"docker.io/mongo:7.0.7"``)
 
 This configuration parameter defines which MongoDB Docker image to use.
 
 .. https://hub.docker.com/_/mysql/tags?page=1&name=8.0
 
-- ``DOCKER_IMAGE_MYSQL`` (default: ``"docker.io/mysql:8.1.0"``)
+- ``DOCKER_IMAGE_MYSQL`` (default: ``"docker.io/mysql:8.4.0"``)
 
 This configuration parameter defines which MySQL Docker image to use.
 
@@ -132,7 +132,7 @@ Open edX customisation
 
 This defines the git repository from which you install Open edX platform code. If you run an Open edX fork with custom patches, set this to your own git repository. You may also override this configuration parameter at build time, by providing a ``--build-arg`` option.
 
-- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/quince.1"``, or ``master`` in :ref:`nightly <nightly>`)
+- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/redwood.1"``, or ``master`` in :ref:`nightly <nightly>`)
 
 This defines the default version that will be pulled from all Open edX git repositories.
 
@@ -378,74 +378,28 @@ Note that your edx-platform version must be a fork of the latest release **tag**
 
 If you don't create your fork from this tag, you *will* have important compatibility issues with other services. In particular:
 
-- Do not try to run a fork from an older (pre-Quince) version of edx-platform: this will simply not work.
+- Do not try to run a fork from an older (pre-Redwood) version of edx-platform: this will simply not work.
 - Do not try to run a fork from the edx-platform master branch: there is a 99% probability that it will fail.
-- Do not try to run a fork from the open-release/quince.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/quince.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/quince.master branch.
+- Do not try to run a fork from the open-release/redwood.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/redwood.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/redwood.master branch.
 
 .. _i18n:
 
-Adding custom translations
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Getting and customizing Translations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you are not running Open edX in English (``LANGUAGE_CODE`` default: ``"en"``), chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/openedx/edx-platform/tree/open-release/quince.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
+Tutor builds images with the latest translations using the ```atlas pull`` command <https://github.com/openedx/openedx-atlas>`__.
 
-Tutor offers a relatively simple mechanism to add custom translations to the openedx Docker image. You should create a folder that corresponds to your language code in the "build/openedx/locale" folder of the Tutor environment. This folder should contain a "LC_MESSAGES" folder. For instance::
+By default the translations are pulled from the `openedx-translations repository <https://github.com/openedx/openedx-translations>`_
+from the ``ATLAS_REVISION`` branch. You can use custom translations on your fork of the openedx-translations repository by setting the following configuration parameters:
 
-    mkdir -p "$(tutor config printroot)/env/build/openedx/locale/fr/LC_MESSAGES"
+- ``ATLAS_REVISION`` (default: ``"main"`` on nightly and ``"{{ OPENEDX_COMMON_VERSION }}"`` if a named release is used)
+- ``ATLAS_REPOSITORY`` (default: ``"openedx/openedx-translations"``). There's a feature request to `support GitLab and other providers <https://github.com/openedx/openedx-atlas/issues/20>`_.
+- ``ATLAS_OPTIONS`` (default: ``""``) Pass additional arguments to ``atlas pull``. Refer to the `atlas documentations <https://github.com/openedx/openedx-atlas>`_ for more information.
 
-The language code should be similar to those used in edx-platform or openedx-i18n (see links above).
-
-Then, add a "django.po" file there that will contain your custom translations::
-
-    msgid ""
-    msgstr ""
-    "Content-Type: text/plain; charset=UTF-8"
-
-    msgid "String to translate"
-    msgstr "你翻译的东西 la traduction de votre bidule"
-
-
-.. warning::
-    Don't forget to specify the file ``Content-Type`` when adding message strings with non-ASCII characters; otherwise a ``UnicodeDecodeError`` will be raised during compilation.
-
-The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/openedx/edx-platform/blob/open-release/quince.master/conf/locale/en/LC_MESSAGES/django.po>`__.
-
-If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/openedx/edx-platform/blob/open-release/quince.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
-
-To recap, here is an example. To translate a few strings in French, both from django.po and djangojs.po, we would have the following file hierarchy::
-
-    $(tutor config printroot)/env/build/openedx/locale/
-        fr/
-            LC_MESSAGES/
-                django.po
-                djangojs.po
-
-With django.po containing::
-
-    msgid ""
-    msgstr ""
-    "Content-Type: text/plain; charset=UTF-8"
-
-    msgid "It works! Powered by Open edX{registered_trademark}"
-    msgstr "Ça marche ! Propulsé by Open edX{registered_trademark}"
-
-And djangojs.po::
-
-    msgid ""
-    msgstr ""
-    "Content-Type: text/plain; charset=UTF-8"
-
-    msgid "%(num_points)s point possible (graded, results hidden)"
-    msgid_plural "%(num_points)s points possible (graded, results hidden)"
-    msgstr[0] "%(num_points)s point possible (noté, résultats cachés)"
-    msgstr[1] "%(num_points)s points possibles (notés, résultats cachés)"
-
-Then you will have to re-build the openedx Docker image::
-
-    tutor images build openedx
-
-Beware that this will take a long time! Unfortunately, it's difficult to accelerate this process, as translation files need to be compiled before collecting the assets. In development it's possible to accelerate the iteration loop -- but that exercise is left to the reader.
+If you are not running Open edX in English (``LANGUAGE_CODE`` default: ``"en"``), chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens!
 
+With ``atlas``, it's possible to add custom translations by either `contributing to the Translations project in Transifex <https://docs.openedx.org/en/latest/translators/index.html>`_ or forking the `openedx-translations repository <https://github.com/openedx/openedx-translations>`_
+and making custom changes as explained in `the repository docs <https://github.com/openedx/openedx-translations#readme>`_.
 
 Running a different ``openedx`` Docker image
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/configuration.rst

@@ -99,9 +99,9 @@ To open a python shell in the LMS or CMS, run::
 
 You can then import edx-platform and django modules and execute python code.
 
-To rebuild assets, you can use the ``openedx-assets`` command that ships with Tutor::
+To rebuild assets, you can run the ``build-dev`` NPM script that comes with edx-plaform::
 
-    tutor dev run lms openedx-assets build --env=dev
+    tutor dev run lms npm run build-dev
 
 
 .. _specialized for developer usage:
@@ -113,7 +113,7 @@ The ``openedx-dev`` Docker image is based on the same ``openedx`` image used by
 
 - The user that runs inside the container has the same UID as the user on the host, to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository).
 - Additional Python and system requirements are installed for convenient debugging: `ipython <https://ipython.org/>`__, `ipdb <https://pypi.org/project/ipdb/>`__, vim, telnet.
-- The edx-platform `development requirements <https://github.com/openedx/edx-platform/blob/open-release/quince.master/requirements/edx/development.in>`__ are installed.
+- The edx-platform `development requirements <https://github.com/openedx/edx-platform/blob/open-release/redwood.master/requirements/edx/development.in>`__ are installed.
 
 
 If you are using a custom ``openedx`` image, then you will need to rebuild ``openedx-dev`` every time you modify ``openedx``. To so, run::

docs/dev.rst

@@ -123,11 +123,11 @@ Major Open edX releases are published twice a year, in June and December, by the
 4. Test the new release in a sandboxed environment.
 5. If you are running edx-platform, or some other repository from a custom branch, then you should rebase (and test) your changes on top of the latest release tag (see :ref:`edx_platform_fork`).
 
-The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``launch`` command (see above). The single difference is that if the ``launch`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``launch``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Palm to Quince and rebuild some Docker images, run::
+The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``launch`` command (see above). The single difference is that if the ``launch`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``launch``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Quince to Redwood and rebuild some Docker images, run::
 
     tutor config save
     tutor images build all # list the images that should be rebuilt here
-    tutor local upgrade --from=palm
+    tutor local upgrade --from=quince
     tutor local launch
 
 .. _autocomplete:

docs/install.rst

@@ -22,7 +22,7 @@ Yes :) This is what happens when you run ``tutor local launch``:
 2. Configuration files are generated from templates.
 3. Docker images are downloaded.
 4. Docker containers are provisioned.
-5. A full, production-ready Open edX platform (`Quince <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-quince.master/platform_releases/quince.html>`__ release) is run with docker-compose.
+5. A full, production-ready Open edX platform (`Redwood <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-redwood.master/platform_releases/redwood.html>`__ release) is run with docker-compose.
 
 The whole procedure should require less than 10 minutes, on a server with good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary.
 

docs/quickstart.rst

@@ -7,10 +7,10 @@ Plugin indexes are a great way to have your plugins discovered by other users. P
 Index file paths
 ================
 
-A plugin index is a yaml-formatted file. It can be stored on the web or on your computer. In both cases, the index file location must end with "<current release name>/plugins.yml". For instance, the following are valid index locations if you run the Open edX "Quince" release:
+A plugin index is a yaml-formatted file. It can be stored on the web or on your computer. In both cases, the index file location must end with "<current release name>/plugins.yml". For instance, the following are valid index locations if you run the Open edX "Redwood" release:
 
-- https://overhang.io/tutor/main/quince/plugins.yml
-- ``/path/to/your/local/index/quince/plugins.yml``
+- https://overhang.io/tutor/main/redwood/plugins.yml
+- ``/path/to/your/local/index/redwood/plugins.yml``
 
 To add either indexes, run the ``tutor plugins index add`` command without the suffix. For instance::
 
@@ -106,9 +106,9 @@ Manage plugins in development
 Plugin developers and maintainers often want to install local versions of their plugins. They usually achieve this with ``pip install -e /path/to/tutor-plugin``. We can improve that workflow by creating an index for local plugins::
 
     # Create the plugin index directory
-    mkdir -p ~/localindex/quince/
+    mkdir -p ~/localindex/redwood/
     # Edit the index
-    vim ~/localindex/quince/plugins.yml
+    vim ~/localindex/redwood/plugins.yml
 
 Add the following to the index::
 

docs/reference/indexes.rst

@@ -158,7 +158,9 @@ The error produced should help better understand what is happening.
 The chosen default language does not display properly
 -----------------------------------------------------
 
-By default, Open edX comes with a `limited set <https://github.com/openedx/edx-platform/blob/master/conf/locale/config.yaml>` of translation/localization files. To complement these languages, we add locales from the `openedx-i18n project <https://github.com/openedx/openedx-i18n/blob/master/edx-platform/locale/config-extra.yaml>`_. But not all supported locales are downloaded. In some cases, the chosen default language will not display properly because it was not packaged in either edx-platform or openedx-i18n. If you feel like your language should be packaged, please `open an issue on the openedx-i18n project <https://github.com/openedx/openedx-i18n/issues>`_.
+By default, Open edX comes with a `limited set <https://github.com/openedx/openedx-translations/tree/main/translations/edx-platform/conf/locale>` of translation/localization files.
+
+Refer to the :ref:`i18n` section for more information about using your own translations.
 
 When I make changes to a course in the CMS, they are not taken into account by the LMS
 --------------------------------------------------------------------------------------