[DPE-3573][DPE-3579] Fix #164 - check _cluster/settings instead of opensearch.yml for plugins

[phvalguima]

Adds a GET /_cluster/settings, which returns the configuration of the cluster at the moment, including defaults. This way, plugins can check if they should add their configs and request a restart during the start cycle of the charm.

The _is_enabled is also extended to use the cluster settings instead of checking the configuration files directly.

Moved OpenSearchPluginConfig to extend pydantic.BaseModel. It ensures that configuration is translated to Dict[str, str], in special the boolean value, that must have a format of: true or false.

Also closes DPE-3579: move the plugin settings status to use the charm's status.{set,clear} APIs.

[Mehdi-Bendriss]

It will be hard to track any side effects introduced by these changes, let's split this PR in 4 separate PRs please:

1. plugins: rely on _cluster/settings instead of opensearch.yml for plugins:
   • including: prevent restarting randomly at the start of the cluster
   • PR-175 depends on / waits for this fix
   • some tests on the data integrator fail because of it
2. experiment with synchronous start: this should only be about making the _start_opensearch synchronous (we haven't confirmed this is solving the issue and the impact it has):
   • no deferrals
   • increase of the timeout duration on self.opensearch.start
   • managing the OpenSearchStartTimeoutError, OpenSearchNotFullyReadyError - how to handle in a synchronous scenario?
3. the restart part (i'm not entirely sure why we need it?)
4. the rollingops part should be out of the charm - and if changes need to be made, they should be done upstream

I have a couple of questions:

• why do we need to go through the restart method for anything start related? it seems we are complicating the logic of the restart by making it a single point of entry. (please refer to PR-175 for ideas)
• I'm not sure I understand what we're trying to solve in the changes in the _on_start hook? I re-read the previous logic, and it seems it was correct as it was prior to the new changes

[phvalguima]

To clarify: start logic for me is all the logic directly involved in getting the service up and running for the very first time.

Regarding the questions:

why do we need to go through the restart method for anything start related? it seems we are complicating the logic of the restart by making it a single point of entry. (please refer to PR-175 for ideas)

There are some reasons:

1. The charm-rolling-ops never checks directly for the callback method in the event itself. It sets the callback_override in the relation and then, in a next RunWithLockEvent, it searches in the relation databag for the override. So, if we switch methods, we may have:

• Hook 1, Event 1: Unit runs: acquire_lock.emit(callback_override=_start_opensearch)
• Hook 1, Event 1: Unit finishes Event 1
  ...
• Hook 1, Event X: Unit runs: acquire_lock.emit(callback_override=_restart_opensearch) <--- we have overriden the value in the relation databag
  ...
• Hook N, Event 1: Unit received "RunWithLockEvent" and recovers the callback: _restart_opensearch is called
  There is a potential race condition here

2. The start logic is still being executed several hooks later, for each unit. That increases the chance of overlapping _start_opensearch and _restart_opensearch

I'm not sure I understand what we're trying to solve in the changes in the _on_start hook? I re-read the previous logic, and it seems it was correct as it was prior to the new changes

For some reasons: #154 - I find the start logic in the charm has a lot going on. It needs information that only comes later, like certificates. That means we have some "mixing", where certain events of the lifecycle have to happen even before the actual start. That means: we need a way to defer.
The first step: the _on_start gets more of the logic, which also cleans up the _start_opensearch and allows us to have one single callback_override: _restart_opensearch. That first simplification moves some of the defer up to the _on_start. Then, I still have to deal, as you said, with delayed connections. You can see I added tenacity and I have several retrials in the start/stop. Now, these are synchronous.
I cannot make the tests work 100% with simply removing the extra "restart" caused by plugin-manager originally - it randomly fails by never receiving the lock.

---

Regarding your points:

1. plugins: repurposed the PR back to focus only in the plugin

2. experiment with sync start: yes, that was my first trial. But then I noticed it did not resolve the problem - our starting cycle has several stages and we need some way to defer the start until we are ready. That is where I started to work in the RollingOpsManager instead. I am trying to have a retriable locking system.

3. As mentioned above, I do not want to accidentally mix _start_opensearch and _restart_opensearch. Given a start is just a "restart without stopping", I see no reason to merge both methods.

4. Ack, done in: charm-rolling-ops#12. That was my 1st iteration on the problem. I will prepare a PR using OpenSearch for the testing. Re:PR#175, if you check the rolling-ops change, I think we should extend "WorkloadLockManager" and overload "can_node_be_safe_stopped", where it would check the output of "wait_for_shards_relocation" to see if it is safe to stop that node.

[phvalguima]

The backup error in: https://github.com/canonical/opensearch-operator/actions/runs/7951972946/job/21713806527, is a status cleanup addressed in #186

[phvalguima]

The h-scaling-integration failed with a time out:

2024-02-19T19:14:36.5332986Z �[32mINFO    �[0m tests.integration.helpers_deployments:helpers_deployments.py:271 	App: opensearch - Units - conditions unmet.
2024-02-19T19:14:36.5334811Z �[32mINFO    �[0m tests.integration.helpers_deployments:helpers_deployments.py:272 
2024-02-19T19:14:36.5335427Z 	app: opensearch active -- message: None
2024-02-19T19:14:36.5335956Z 		opensearch-3  -- (10.171.13.159) -- [idle (since: 18:44:39)] active: 
2024-02-19T19:14:36.5336557Z 		opensearch-5  -- (10.171.13.194) -- [idle (since: 18:44:39)] active: 
2024-02-19T19:14:36.5337257Z 		opensearch-6  -- (10.171.13.245) -- [idle (since: 18:44:37)] waiting: Awaiting service operation
2024-02-19T19:14:36.5337962Z 		opensearch-7  -- (10.171.13.183) -- [idle (since: 18:44:39)] active: 
2024-02-19T19:14:36.5338550Z 		opensearch-8* -- (10.171.13.236) -- [idle (since: 18:44:48)] active: 
2024-02-19T19:14:36.5338885Z 
2024-02-19T19:14:36.5339362Z �[1m�[31mERROR   �[0m tests.integration.helpers_deployments:helpers_deployments.py:346 wait_until -- Timed out!

[Mehdi-Bendriss]

Thanks Pedro - I left some questions and comments

The base branch was changed.