feat(controller, e2e): Refactor reconcile loop and add E2E tests for scaling and promotion

[ballista01]

Summary

This pull request introduces two main improvements:

1. A refactoring of the EtcdClusterReconciler's main Reconcile loop to improve its structure and testability.
2. The addition of new End-to-End (E2E) tests to cover critical cluster lifecycle scenarios, such as scaling and member promotion.

The E2E test additions are motivated by the need to validate controller logic that depends on a functional Kubernetes environment with DNS and networking, which is not feasible at the unit test level without privileged operations (e.g., /etc/hosts modification).

1. Reconciliation Loop Refactoring (internal/controller/etcdcluster_controller.go)

The primary Reconcile function has been decomposed into four distinct, sequential phases, each with a clear responsibility:

• fetchAndValidateState: Retrieves and validates the primary and associated resources.
• syncPrimitives: Ensures the existence of foundational Kubernetes objects (StatefulSet, Service).
• performHealthChecks: Gathers health and membership status from the etcd cluster.
• reconcileClusterState: Executes the core scaling and membership management logic.

A reconcileState struct is now used to pass the collected state through these phases, standardizing the data flow within a single reconciliation cycle.

2. E2E Test Suite Additions (test/e2e/)

New E2E tests have been added to verify the operator's behavior in a kind environment.

New Test Scenarios:

• TestScaleUpFrom1To3: Verifies the process of scaling a cluster from 1 to 3 members.
• TestScaleDownFrom3To1: Verifies the process of scaling a cluster from 3 to 1 member.
• TestPromoteReadyLearner: Verifies the operator's ability to handle a cluster with a pre-existing learner member. It confirms the operator can bring the cluster to the desired state (spec.size: 3) and successfully promote the learner.

Testing Framework Enhancements:

• A new test/e2e/helpers.go file has been introduced to provide reusable test utilities.
• A key helper, execInPod, is included. This function allows tests to execute etcdctl commands directly within the etcd pods to assert against the internal state of the etcd cluster, such as the output of etcdctl member list.

3. Evolution of Unit Tests (internal/controller/etcdcluster_controller_test.go)

• The unit tests that previously required sudo privileges for /etc/hosts modifications have been superseded.
• Unit tests that validate specific internal logic and error handling without requiring network hacks (e.g., TestWaitForUnreadyLearner which simulates a lagging learner) have been preserved, as they cover edge cases that are difficult to reproduce reliably in an E2E environment.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ballista01
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @ballista01. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jberkus]

/ok-to-test