feat: add atlas pull support - FC-0012

[shadinaif]

Add atlas pull support. (atlas pull) will be performed only if OPENEDX_ATLAS_PULL environment variable is set

• Tested on devstack, docker build is working fine
• Tested on devstack, make extract_translation is still running fine with latest i18-tools==1.3.0
• Tested on devstack, OPENEDX_ATLAS_PULL=yes make pull_translation is working fine
• Tested on devstack, OPENEDX_ATLAS_PULL=yes make pull_translation is deleting old translations and pulling the latest translations

References

This pull request is part of the FC-0012 project which is sparked by the Translation Infrastructure update OEP-58.

Check the links above for full information about the overall project.

Internalization is being rearchitected in Open edX Python, XBlock, Micro-frontend, and other projects. There are a number of immediately visible changes:

• Remove source and language translations from the repositories, hence no .json, .po or .mo files will be committed into the repos.
• Add standardized make extract_translations in all repositories
• Push user-facing messages strings into openedx/openedx-translations.
• Integrate root repositories with openedx/openedx-atlas to pull translations on build/deploy time

Breaking Changes

One of the primary goals of the project is to avoid breaking changes. If you noticed any suspicious code, please raise your concern. But before that, please know the strategy we're following to avoid breaking changes

For XBlocks:

• The standard translation path must be conf/locale. Therefore, we are creating a link from conf/locale to translations so Atlas can find the path without disrupting the current way of having translations locally within the XBlocks
• openedx-translations will have a related PR that adds the XBlock to the pipeline. This will not affect the current way of managing/updating translations
• At the end of the project, a clear change log will be added and all translations will be handled by Atlas. Thus, the local translation will be removed from the repo within the version bump
• A notification for the community will be published to ensure that everyone knows why translations directories are removed from all repos

[openedx-webhooks]

Thanks for the pull request, @shadinaif! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[shadinaif]

please review @OmarIthawi , @brian-smith-tcril

[brian-smith-tcril]

@shadinaif looks like the CI failed

[shadinaif]

Fixed. Please run tests again @OmarIthawi , @brian-smith-tcril

[brian-smith-tcril]

@colinbrash you are listed as "review required from" on the spreadsheet for this repo. I'm hoping to get this merged soon, thanks!

[OmarIthawi]

@shadinaif I've tested it locally and it worked except for the compilemssage step

Do you mind checking the error bellow?

$ docker build -t temp-ecommerce .
$ docker run -it docker.io/library/temp-ecommerce bash
# atlas --version
v0.5.0
# make OPENEDX_ATLAS_PULL=yes pull_translations

Error:

  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/edx/app/ecommerce/ecommerce/ecommerce/settings/devstack.py", line 6, in <module>
    from ecommerce.settings.production import *
  File "/edx/app/ecommerce/ecommerce/ecommerce/settings/production.py", line 60, in <module>
    with codecs.open(CONFIG_FILE, encoding='utf-8') as f:
  File "/usr/lib/python3.8/codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: '/edx/etc/ecommerce.yml'
make: *** [Makefile:134: pull_translations] Error 1

[shadinaif]

Thank you for the tests 😃 @OmarIthawi

It's running fine after copying the file from devstack repo to /edx/etc/ecommerce.yml inside the docker container. The file does not exist by default. Check the instructions here. It's set by devstack here

[OmarIthawi]

I'll test again and let you know.

[OmarIthawi]

It works on devstack. Thanks @shadinaif!!

[shadinaif]

Hi @brian-smith-tcril , I've rebased on master, please run tests and merge

[shadinaif]

rebased again

[openedx-webhooks]

@shadinaif 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.