fix: WP-813 CMS customization fails, CMS is old #1034

wesleyboar · 2024-12-09T17:47:43Z

Overview & Changes

Update CMS settings & secrets documentation.
Document CMS customization.
Update CMS sample secrets to mimic TACC/Core-CMS's.
Update CMS image.

Status

Migrate CMS settings to one settings/ directory.
+ Update other repositories accordingly.

Testing & UI

CMS Settings & Secrets

If Core-Portal is running locally, run make stop.
Backup your CMS settings & secrets.
^{e.g. cp server/conf/cms/ server/conf/cms-before-WP-813/}

Add CMS settings via new instructions e.g.

curl "https://cdn.jsdelivr.net/gh/TACC/Core-CMS@main/taccsite_cms/secrets.example.py" -o server/conf/cms/settings_custom.py
cp server/conf/cms/secrets.sample.py server/conf/cms/secrets.py
curl "https://cdn.jsdelivr.net/gh/TACC/Core-CMS@main/taccsite_cms/settings_local.example.py" -o server/conf/cms/settings_local.py

Run make start.
Verify no errors and these —
- server/conf/cms/settings_custom.py
- server/conf/cms/secrets.py
- server/conf/cms/settings_local.py
— exist as files not directories.
Verify Core-Portal's Core-CMS loads in browser.

CMS Settings

CMS Customization via Settings

Uncomment these lines in settings_local.py —

# PORTAL_IS_TACC_CORE_PORTAL = False
# PORTAL_HAS_LOGIN = False

Reload Core-CMS in browser.
Verify "Login" link / Portal nav is absent.

WP-813.mov

CMS Version

Open browser's Developer Tools on CMS page.
In "Network" tab, open core-syles.*.css content.
At top of file, verify version is latest Core-CMS release.

CMS version

codecov · 2024-12-09T17:51:12Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.68%. Comparing base (229d1cb) to head (a4538a0).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1034   +/-   ##
=======================================
  Coverage   72.68%   72.68%           
=======================================
  Files         534      534           
  Lines       33657    33657           
  Branches     2989     2989           
=======================================
  Hits        24464    24464           
  Misses       8995     8995           
  Partials      198      198

Flag	Coverage Δ
javascript	`75.40% <ø> (ø)`
unittests	`60.84% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

…e-Portal into enhance/CMD-194-update-core-cms

server/conf/cms/secrets.sample.py

wesleyboar · 2024-12-09T18:53:40Z

server/conf/docker/docker-compose-dev.all.debug.yml

+      - ../cms/settings_local.py:/code/taccsite_cms/settings_local.py
+      - ../cms/settings_custom.py:/code/taccsite_cms/settings_custom.py


This allows settings_local.py and settings_custom.py to function. I did not add :ro, because (based on my chat with Docker's AI) that will not prevent directory from showing if files exist.

Details

These are methods to prevent directory from being created if single-file volume is missing:

A. Name the volume. Set it as external: true; this causes error instead of empty directory.

B. Ensure the files exist before composing the Docker environment.

In Production the :ro ensures that when a new portal VM is provisioned and configured if any of the placed files are forgotten (secrets, settings_local) the portal's docker engine will not generate an empty directory and try to mount said empty dir, docker will instead complain about the missing files - which is what we want. You may still want the mount to fail in local dev if the files are not present (ie - were not curled or copied as needed)? Its a quick way of knowing you missed something. Thoughts?

A. Sounds fine, but adding :ro does not have that effect for me locally. Are my test steps flawed?

Test Steps

Add the :ro to each settings & secrets file volume.

Run make stop.

Deleted my existing settings & secrets files.

Run make start.

❌ See empty directories added.

b11800a

B. I tried using verbose syntax, so I could change the mount type to bind. This change, with read_only: true worked i.e. caused error if a file is missing.

Test Steps

Add the :ro to each settings & secrets file volume.

Run make stop.

Deleted my existing settings & secrets directories.
^{(from testing method A)}

Run make start.

✅ No empty directories added.

ℹ️ Console reports error about missing file.

59a72f8

C. I investigated again the other method I'd read, a named volume with external: true. It requires manually adding a volume first and an equally verbose docker-compose file change. And I don't think it will work (because how would container tell CMS where file is).

Instructions (Untested)

First, create a named volume manually (if it doesn't already exist):
docker volume create my_volume

Then, in your Docker Compose file, you can use this volume and mark it as external:
services: your_service: image: your_image volumes: - my_volume:/path/in/container/file.txt volumes: my_volume: external: true

@rstijerina & @taoteg, I do not want to start Idea D yet.¹

This PR, as of latest commit 4c0fb05, would create directories. But, anyone who would customize the CMS, is instructed to delete them, touch the files, restart, and populate the settings as necessary.

Is this sufferable for now?

Footnotes

Because it requires refactor in Core-CMS, clients and developers to move their settings, and Camino to update where it places CMS settings. ↩

How about your verbose method, that doesn't create empty dirs?

B.2. Ew. Okay. See 985e86a. But that causes an error if file does not exist… I thought new goal was to not create files unless necessary.

E. To avoid error and directory and file content, create empty files. Proposal: I add back "CMS" section to README.md but with new command. See 985e86a...3c0187bb.

I'm fine with [any…] just mentioning drawbacks.

—

P.S. [Devs have so many commands] to run… couldn't some be moved to a script that Docker runs on up?

Also, if no rush on this, it can stay open to remind me to do Idea D.

Thanks for background again. Idea D is best imo

server/conf/cms/secrets.sample.py

README.md

#1034 (comment)

Previous attempt to error if files are missing failed: #1034 (comment) This method succeeded, an error occurred: > invalid mount config for type "bind": bind source path does not exist: /host_mnt/[…]/server/conf/cms/secrets.py

server/conf/docker/docker-compose-dev.all.debug.yml

rstijerina · 2024-12-09T21:06:05Z

README.md

+```sh
+curl "https://cdn.jsdelivr.net/gh/TACC/Core-CMS@main/taccsite_cms/secrets.example.py" -o server/conf/cms/settings_custom.py
+cp server/conf/cms/secrets.sample.py server/conf/cms/secrets.py
+curl "https://cdn.jsdelivr.net/gh/TACC/Core-CMS@main/taccsite_cms/settings_local.example.py" -o server/conf/cms/settings_local.py
+```


99% of use of the CMS in this repo by devs is just to have a homepage. I think a good goal would be to have a default config (the existing secrets.sample.py) to get up and running immediately, and if need be a dev who needs bespoke CMS settings can set them after the fact. Think we can figure something out?

I had thought so, and am fine with minimal working secrets (as it was).

@taoteg reports devs adding CMS non-secret settings to CMS secrets.py. Or maybe I misunderstood. For fear that is happening:

I added support for settings_custom.py and settings_local.py.

To avoid settings_custom.py and settings_local.py being added as directories if they are absent, I added commands to create them.

To avoid needing to keep those files up-to-date, I curled them from TACC/Core-CMS.

Personally I think keeping a working set of default settings committed makes more sense for local dev, and updates as needed can be committed to the repo.

How about this: We commit a set of local settings that are committed with dummy credentials that will be imported into the CMS settings on make start, and if a (untracked) secrets.py happens to exist, then that acts as an override

I am looking for parity between env setups here. We should be using the same config files (settings, settings_custom, settings_local, secrets) in every env (dev, local, pprd, prod).

By leaving out parts of the default config structure we are not training devs on the expected practice for how these files exist in pprd/prod and why certain settings exist in certain files.

Examples:

There is no value for SILENCED_SYSTEM_CHECKS = ['captcha.recaptcha_test_key_error'] in the example settings. This error will creep up in the logging (annoying). [Supposedly this will go away when FP-1728: Remove once servers have RECAPTCHA_..._KEY settings but so far has not for me.. maybe due to the old CMS image?]. This setting should go in settings_local, but how would you do that in the CMS with the current setup? By overloading the secrets file... which is bad practice.

This request is as much about consistency in workflows as it is about having good settings out of the box that use a current CMS image.

Ah, the SILENCED_SYSTEM_CHECKS is in Core-CMS settings_local.py…. Another way to resolve that missing piece is add it to a committed and tracked settings_custom.py (which @rstijerina proposed to us privately).

New solution available via settings_default.py (which Core-CMS supports in TACC/Core-CMS#904.

Verbose config way to stop creation of empty directories. Drawback: Error if files do not exist.

Simple manual way to stop creation of empty directories. Drawback: Manual step… BUT it replaces step that had been here.

server/conf/cms/settings_default.py

wesleyboar added 2 commits December 9, 2024 11:35

fix: CMD-194 CMS customization fails, CMS is old

5472b9d

fix: add missing step to fix settings_… dirs

9d91f9e

Merge branch 'main' into enhance/CMD-194-update-core-cms

f8e07b8

wesleyboar mentioned this pull request Dec 9, 2024

feat: cmd-194 enable portal nav by default TACC/Core-CMS#903

Merged

wesleyboar requested review from taoteg and rstijerina December 9, 2024 18:42

wesleyboar added 3 commits December 9, 2024 12:45

docs: CMD-194 clearer statement

62c286b

Merge branch 'enhance/CMD-194-update-core-cms' of github.com:TACC/Cor…

0faa912

…e-Portal into enhance/CMD-194-update-core-cms

docs: CMD-194 remvoe extra line

fa70269

wesleyboar commented Dec 9, 2024

View reviewed changes

wesleyboar added 2 commits December 9, 2024 14:43

enhance: CMD-194 add :ro like production

b11800a

#1034 (comment)

fix: CMD-194 :ro → type:bind, read_only:true

59a72f8

Previous attempt to error if files are missing failed: #1034 (comment) This method succeeded, an error occurred: > invalid mount config for type "bind": bind source path does not exist: /host_mnt/[…]/server/conf/cms/secrets.py

rstijerina reviewed Dec 9, 2024

View reviewed changes

refactor: setings_default.py

3e299d0

wesleyboar mentioned this pull request Dec 9, 2024

feat: default custom settings settings_default TACC/Core-CMS#904

Merged

wesleyboar added 7 commits December 9, 2024 17:50

test: CMS suppoting settings_default.py

35e0a36

docs(README): CMS section becomes minor ref. link

a5379e9

docs: add missing word

4c0fb05

fix: settings dirs, type:bind read_only:true

985e86a

Verbose config way to stop creation of empty directories. Drawback: Error if files do not exist.

fix: settings dirs via manual command

699334f

Simple manual way to stop creation of empty directories. Drawback: Manual step… BUT it replaces step that had been here.

docs: fix typo in command

3c0187b

core: latest CMS image

a4538a0

wesleyboar added bug Something isn't working enhancement New feature or request labels Dec 10, 2024

wesleyboar commented Dec 10, 2024

View reviewed changes

server/conf/cms/settings_default.py Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: WP-813 CMS customization fails, CMS is old #1034

fix: WP-813 CMS customization fails, CMS is old #1034

wesleyboar commented Dec 9, 2024 •

edited

Loading

codecov bot commented Dec 9, 2024 •

edited

Loading

wesleyboar Dec 9, 2024

taoteg Dec 9, 2024

wesleyboar Dec 9, 2024 •

edited

Loading

wesleyboar Dec 9, 2024 •

edited

Loading

wesleyboar Dec 9, 2024

wesleyboar Dec 10, 2024

rstijerina Dec 10, 2024

wesleyboar Dec 10, 2024 •

edited

Loading

wesleyboar Dec 10, 2024

rstijerina Dec 10, 2024

rstijerina Dec 9, 2024

wesleyboar Dec 9, 2024

rstijerina Dec 9, 2024

taoteg Dec 9, 2024 •

edited

Loading

wesleyboar Dec 9, 2024

wesleyboar Dec 10, 2024 •

edited

Loading

		- ../cms/settings_local.py:/code/taccsite_cms/settings_local.py
		- ../cms/settings_custom.py:/code/taccsite_cms/settings_custom.py

fix: WP-813 CMS customization fails, CMS is old #1034

Are you sure you want to change the base?

fix: WP-813 CMS customization fails, CMS is old #1034

Conversation

wesleyboar commented Dec 9, 2024 • edited Loading

Overview & Changes

Status

Related

Testing & UI

CMS Settings & Secrets

CMS Customization via Settings

CMS Version

codecov bot commented Dec 9, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesleyboar Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

wesleyboar Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Footnotes

Choose a reason for hiding this comment

wesleyboar Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taoteg Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesleyboar Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

wesleyboar commented Dec 9, 2024 •

edited

Loading

codecov bot commented Dec 9, 2024 •

edited

Loading

wesleyboar Dec 9, 2024 •

edited

Loading

wesleyboar Dec 9, 2024 •

edited

Loading

wesleyboar Dec 10, 2024 •

edited

Loading

taoteg Dec 9, 2024 •

edited

Loading

wesleyboar Dec 10, 2024 •

edited

Loading