Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: WP-813 CMS customization fails, CMS is old #1034

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

fix: WP-813 CMS customization fails, CMS is old #1034

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
5472b9d
fix: CMD-194 CMS customization fails, CMS is old
wesleyboar Dec 9, 2024
9d91f9e
fix: add missing step to fix settings_… dirs
wesleyboar Dec 9, 2024
f8e07b8
Merge branch 'main' into enhance/CMD-194-update-core-cms
wesleyboar Dec 9, 2024
62c286b
docs: CMD-194 clearer statement
wesleyboar Dec 9, 2024
0faa912
Merge branch 'enhance/CMD-194-update-core-cms' of github.com:TACC/Cor…
wesleyboar Dec 9, 2024
fa70269
docs: CMD-194 remvoe extra line
wesleyboar Dec 9, 2024
b11800a
enhance: CMD-194 add `:ro` like production
wesleyboar Dec 9, 2024
59a72f8
fix: CMD-194 `:ro` → `type:bind`, `read_only:true`
wesleyboar Dec 9, 2024
3e299d0
refactor: setings_default.py
wesleyboar Dec 9, 2024
35e0a36
test: CMS suppoting settings_default.py
wesleyboar Dec 9, 2024
a5379e9
docs(README): CMS section becomes minor ref. link
wesleyboar Dec 10, 2024
4c0fb05
docs: add missing word
wesleyboar Dec 10, 2024
985e86a
fix: settings dirs, `type:bind` `read_only:true`
wesleyboar Dec 10, 2024
699334f
fix: settings dirs via manual command
wesleyboar Dec 10, 2024
3c0187b
docs: fix typo in command
wesleyboar Dec 10, 2024
a4538a0
core: latest CMS image
wesleyboar Dec 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,11 +107,15 @@ After you clone the repository locally, there are several configuration steps re

##### CMS

- Copy `server/conf/cms/secrets.sample.py` to `server/conf/cms/secrets.py`
Add CMS settings and secrets:

- To emulate a specific CMS project, copy https://github.com/TACC/Core-CMS-Resources/blob/main/__PROJECT_TO_EMULATE__/settings_custom.py to `server/conf/cms/settings_custom.py`
```sh
curl "https://cdn.jsdelivr.net/gh/TACC/Core-CMS@main/taccsite_cms/secrets.example.py" -o server/conf/cms/settings_custom.py
cp server/conf/cms/secrets.sample.py server/conf/cms/secrets.py
curl "https://cdn.jsdelivr.net/gh/TACC/Core-CMS@main/taccsite_cms/settings_local.example.py" -o server/conf/cms/settings_local.py
```
wesleyboar marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

99% of use of the CMS in this repo by devs is just to have a homepage. I think a good goal would be to have a default config (the existing secrets.sample.py) to get up and running immediately, and if need be a dev who needs bespoke CMS settings can set them after the fact. Think we can figure something out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had thought so, and am fine with minimal working secrets (as it was).

@taoteg reports devs adding CMS non-secret settings to CMS secrets.py. Or maybe I misunderstood. For fear that is happening:

  1. I added support for settings_custom.py and settings_local.py.
  2. To avoid settings_custom.py and settings_local.py being added as directories if they are absent, I added commands to create them.
  3. To avoid needing to keep those files up-to-date, I curled them from TACC/Core-CMS.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I think keeping a working set of default settings committed makes more sense for local dev, and updates as needed can be committed to the repo.

How about this: We commit a set of local settings that are committed with dummy credentials that will be imported into the CMS settings on make start, and if a (untracked) secrets.py happens to exist, then that acts as an override

Copy link
Collaborator

@taoteg taoteg Dec 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am looking for parity between env setups here. We should be using the same config files (settings, settings_custom, settings_local, secrets) in every env (dev, local, pprd, prod).

By leaving out parts of the default config structure we are not training devs on the expected practice for how these files exist in pprd/prod and why certain settings exist in certain files.

Examples:

  • There is no value for SILENCED_SYSTEM_CHECKS = ['captcha.recaptcha_test_key_error'] in the example settings. This error will creep up in the logging (annoying). [Supposedly this will go away when FP-1728: Remove once servers have RECAPTCHA_..._KEY settings but so far has not for me.. maybe due to the old CMS image?]. This setting should go in settings_local, but how would you do that in the CMS with the current setup? By overloading the secrets file... which is bad practice.

This request is as much about consistency in workflows as it is about having good settings out of the box that use a current CMS image.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, the SILENCED_SYSTEM_CHECKS is in Core-CMS settings_local.py…. Another way to resolve that missing piece is add it to a committed and tracked settings_custom.py (which @rstijerina proposed to us privately).

Copy link
Member Author

@wesleyboar wesleyboar Dec 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New solution available via settings_default.py (which Core-CMS supports in TACC/Core-CMS#904.


- To override any standard or custom CMS settings, create a `server/conf/cms/settings_local.py`
To populate or customize the CMS, read [Customize a CMS](./docs/customize-cms.md).

#### Build the image for the portal's django container:
make build
Expand Down
26 changes: 26 additions & 0 deletions docs/customize-cms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Customize a CMS

This is all optional. You can develop Core-Portal without CMS customization.

## Basic

To create your own test content, [learn Django CMS as we use it](https://tacc-main.atlassian.net/wiki/x/phdv).

To emulate a specific project's CMS settings:
1. Visit https://github.com/TACC/Core-Portal-Resources/blob/main/.
2. Navigate to specific project.
3. Use its `cms.settings_custom.py` as your `server/conf/cms/settings_custom.py`.

To override any standard or custom CMS settings, edit `server/conf/cms/settings_local.py`.

> **Note:** If your `settings_custom.py` and `settings_local.py` are directories, then they were not properly created before starting environment. To fix:
> 1. Run `make stop`.
> 2. Delete directories named `settings_….py`
> 3. [Add CMS settings.](../README.md#CMS)
> 4. Run `make start`.

## Advanced

To populate content from an existing CMS, follow and adapt instructions to [replicate a CMS database](https://tacc-main.atlassian.net/wiki/x/GwBJAg).

> **Important:** This requires high-level server access or someone to give you a copy of the content.
15 changes: 3 additions & 12 deletions server/conf/cms/secrets.sample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,37 +1,28 @@
########################
# DJANGO SETTINGS
# DJANGO
########################

SECRET_KEY = 'replacethiswithareallysecureandcomplexsecretkeystring'
LOGIN_REDIRECT_URL = '/workbench/dashboard/'

########################
# ELASTICSEARCH
# SEARCH
########################
wesleyboar marked this conversation as resolved.
Show resolved Hide resolved

ES_AUTH = 'username:password'
ES_HOSTS = 'http://elasticsearch:9200'
ES_INDEX_PREFIX = 'cep-dev-{}'
ES_DOMAIN = 'https://cep.test'

es_engine = 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine'
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': es_engine,
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
wesleyboar marked this conversation as resolved.
Show resolved Hide resolved
'URL': ES_HOSTS,
'INDEX_NAME': ES_INDEX_PREFIX.format('cms'),
'KWARGS': {'http_auth': ES_AUTH}
}
}

########################
# RECAPTCHA SETTINGS
########################

RECAPTCHA_PUBLIC_KEY = ''
RECAPTCHA_PRIVATE_KEY = ''
SILENCED_SYSTEM_CHECKS = ['captcha.recaptcha_test_key_error']

wesleyboar marked this conversation as resolved.
Show resolved Hide resolved
########################
# REDMINE TRACKER AUTH
########################
wesleyboar marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
4 changes: 3 additions & 1 deletion server/conf/docker/docker-compose-dev.all.debug.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@
version: "3.8"
services:
cms:
image: taccwma/core-cms:ff737a3
image: taccwma/core-cms:latest
volumes:
- ../cms/secrets.py:/code/taccsite_cms/secrets.py
- ../cms/settings_local.py:/code/taccsite_cms/settings_local.py
- ../cms/settings_custom.py:/code/taccsite_cms/settings_custom.py
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allows settings_local.py and settings_custom.py to function. I did not add :ro, because (based on my chat with Docker's AI) that will not prevent directory from showing if files exist.

Details

These are methods to prevent directory from being created if single-file volume is missing:

  • A. Name the volume. Set it as external: true; this causes error instead of empty directory.
  • B. Ensure the files exist before composing the Docker environment.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Production the :ro ensures that when a new portal VM is provisioned and configured if any of the placed files are forgotten (secrets, settings_local) the portal's docker engine will not generate an empty directory and try to mount said empty dir, docker will instead complain about the missing files - which is what we want. You may still want the mount to fail in local dev if the files are not present (ie - were not curled or copied as needed)? Its a quick way of knowing you missed something. Thoughts?

Copy link
Member Author

@wesleyboar wesleyboar Dec 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A. Sounds fine, but adding :ro does not have that effect for me locally. Are my test steps flawed?

Test Steps
  1. Add the :ro to each settings & secrets file volume.
  2. Run make stop.
  3. Deleted my existing settings & secrets files.
  4. Run make start.
  5. ❌ See empty directories added.
b11800a
b11800ab

Copy link
Member Author

@wesleyboar wesleyboar Dec 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B. I tried using verbose syntax, so I could change the mount type to bind. This change, with read_only: true worked i.e. caused error if a file is missing.

Test Steps
  1. Add the :ro to each settings & secrets file volume.
  2. Run make stop.
  3. Deleted my existing settings & secrets directories.
    (from testing method A)
  4. Run make start.
  5. ✅ No empty directories added.
  6. ℹ️ Console reports error about missing file.
59a72f8
59a72f8e

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C. I investigated again the other method I'd read, a named volume with external: true. It requires manually adding a volume first and an equally verbose docker-compose file change. And I don't think it will work (because how would container tell CMS where file is).

Instructions (Untested)
  1. First, create a named volume manually (if it doesn't already exist): 
    docker volume create my_volume
  2. Then, in your Docker Compose file, you can use this volume and mark it as external: 
    services:
  your_service:
    image: your_image
    volumes:
      - my_volume:/path/in/container/file.txt

volumes:
  my_volume:
    external: true

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rstijerina & @taoteg, I do not want to start Idea D yet.1

This PR, as of latest commit 4c0fb05, would create directories. But, anyone who would customize the CMS, is instructed to delete them, touch the files, restart, and populate the settings as necessary.

Is this sufferable for now?

Footnotes

  1. Because it requires refactor in Core-CMS, clients and developers to move their settings, and Camino to update where it places CMS settings. ↩

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about your verbose method, that doesn't create empty dirs?

Copy link
Member Author

@wesleyboar wesleyboar Dec 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B.2. Ew. Okay. See 985e86a. But that causes an error if file does not exist… I thought new goal was to not create files unless necessary.

E. To avoid error and directory and file content, create empty files. Proposal: I add back "CMS" section to README.md but with new command. See 985e86a...3c0187bb.

I'm fine with [any…] just mentioning drawbacks.

—

P.S. [Devs have so many commands] to run… couldn't some be moved to a script that Docker runs on up?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if no rush on this, it can stay open to remind me to do Idea D.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for background again. Idea D is best imo

- ../cms/uwsgi/uwsgi.ini:/code/uwsgi.ini
- ../../cms/static:/code/static
- ../../cms/media:/code/media
Expand Down