Skip to content

Conversation

ArkaSaha30
Copy link
Contributor

This PR will implement the reconciler for client, peer and server certificate creation for the etcdCluster object and its respective members.

Fixes: #10

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ArkaSaha30
Once this PR has been reviewed and has the lgtm label, please assign hakman for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ArkaSaha30
Copy link
Contributor Author

/test pull-etcd-operator-test-e2e

@ArkaSaha30 ArkaSaha30 force-pushed the cert-manager-reconciler branch 2 times, most recently from dd9729a to fe533db Compare July 3, 2025 09:34
@ArkaSaha30
Copy link
Contributor Author

/test pull-etcd-operator-test-e2e

@ArkaSaha30 ArkaSaha30 force-pushed the cert-manager-reconciler branch 2 times, most recently from 3f702a6 to eba6759 Compare July 7, 2025 13:21
@ArkaSaha30
Copy link
Contributor Author

etcd-operator on  cert-manager-reconciler [$!] via 🐳 colima via 🐹 v1.24.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get pods
NAME                   READY   STATUS    RESTARTS   AGE
etcdcluster-sample-0   1/1     Running   0          2m11s
etcdcluster-sample-1   1/1     Running   0          2m1s
etcdcluster-sample-2   1/1     Running   0          113s

etcd-operator on  cert-manager-reconciler [$] via 🐳 colima via 🐹 v1.24.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get certificates
NAME                              READY   SECRET                            AGE
etcdcluster-sample-0-peer-tls     True    etcdcluster-sample-0-peer-tls     2m
etcdcluster-sample-0-server-tls   True    etcdcluster-sample-0-server-tls   2m5s
etcdcluster-sample-1-peer-tls     True    etcdcluster-sample-1-peer-tls     2m
etcdcluster-sample-1-server-tls   True    etcdcluster-sample-1-server-tls   2m
etcdcluster-sample-2-peer-tls     True    etcdcluster-sample-2-peer-tls     117s
etcdcluster-sample-2-server-tls   True    etcdcluster-sample-2-server-tls   117s
etcdcluster-sample-client-tls     True    etcdcluster-sample-client-tls     2m23s

etcd-operator on  cert-manager-reconciler [$] via 🐳 colima via 🐹 v1.24.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get secrets
NAME                              TYPE                DATA   AGE
etcdcluster-sample-0-peer-tls     kubernetes.io/tls   3      2m5s
etcdcluster-sample-0-server-tls   kubernetes.io/tls   3      2m11s
etcdcluster-sample-1-peer-tls     kubernetes.io/tls   3      2m6s
etcdcluster-sample-1-server-tls   kubernetes.io/tls   3      2m6s
etcdcluster-sample-2-peer-tls     kubernetes.io/tls   3      2m2s
etcdcluster-sample-2-server-tls   kubernetes.io/tls   3      2m3s
etcdcluster-sample-client-tls     kubernetes.io/tls   3      2m28s

@ArkaSaha30 ArkaSaha30 force-pushed the cert-manager-reconciler branch from e37260a to 5e902b6 Compare July 14, 2025 17:56
@ArkaSaha30 ArkaSaha30 force-pushed the cert-manager-reconciler branch from 1b79a53 to 6192aa2 Compare July 23, 2025 15:34
}

func createCertificate(ec *ecv1alpha1.EtcdCluster, ctx context.Context, c client.Client, certName string) error {
cert, certErr := certificate.NewProvider(certificate.ProviderType(ec.Spec.TLS.Provider), c)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ensure provider matches the providerConfig.

@ArkaSaha30 ArkaSaha30 force-pushed the cert-manager-reconciler branch 2 times, most recently from 829112a to 84bb9d8 Compare July 31, 2025 07:01
@ArkaSaha30 ArkaSaha30 changed the title [WIP] Add reconciler for client, peer, server certificate Add reconciler for client, peer, server certificate Jul 31, 2025
@ArkaSaha30 ArkaSaha30 force-pushed the cert-manager-reconciler branch 5 times, most recently from 3027592 to 1d32631 Compare August 4, 2025 10:23
@ivanvc
Copy link
Member

ivanvc commented Aug 5, 2025

Hi @ArkaSaha30, we discussed this in our community meeting.

We would want to have the e2e tests in this pull request too, before we consider it mergeable. Thanks.

@ArkaSaha30
Copy link
Contributor Author

Hi @ArkaSaha30, we discussed this in our community meeting.

We would want to have the e2e tests in this pull request too, before we consider it mergeable. Thanks.

Yes, I'm working on the e2e tests. Will update the PR by EOW.

This commit will add reconciler logic for creating
- client certificate to communicate with the etcdCluster
- server, peer certificate for each of the etcd member communication

Also, mount certificate secrets to member pods

Signed-off-by: ArkaSaha30 <[email protected]>
@ArkaSaha30 ArkaSaha30 force-pushed the cert-manager-reconciler branch from 565b1a7 to 47b635b Compare August 11, 2025 20:08
@ArkaSaha30 ArkaSaha30 requested a review from ahrtr August 12, 2025 05:02
@ArkaSaha30
Copy link
Contributor Author

cc @ivanvc @hakman

@ArkaSaha30
Copy link
Contributor Author

Client, server, peer certificates and their respective secrets:

etcd-operator on  cert-manager-reconciler [$!] via 🐳 colima via 🐹 v1.24.6 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get pods
NAME                   READY   STATUS    RESTARTS   AGE
etcdcluster-sample-0   1/1     Running   0          26s
etcdcluster-sample-1   1/1     Running   0          17s
etcdcluster-sample-2   1/1     Running   0          11s

etcd-operator on  cert-manager-reconciler [$!] via 🐳 colima via 🐹 v1.24.6 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get certificates
NAME                            READY   SECRET                          AGE
etcdcluster-sample-client-tls   True    etcdcluster-sample-client-tls   36s
etcdcluster-sample-peer-tls     True    etcdcluster-sample-peer-tls     36s
etcdcluster-sample-server-tls   True    etcdcluster-sample-server-tls   36s

etcd-operator on  cert-manager-reconciler [$!] via 🐳 colima via 🐹 v1.24.6 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get secrets
NAME                            TYPE                DATA   AGE
etcdcluster-sample-client-tls   kubernetes.io/tls   3      40s
etcdcluster-sample-peer-tls     kubernetes.io/tls   3      39s
etcdcluster-sample-server-tls   kubernetes.io/tls   3      40s

Server and peer secret are mounted to each of the members:

etcd-operator on  cert-manager-reconciler [$!] via 🐳 colima via 🐹 v1.24.6 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k describe pod etcdcluster-sample-0
Name:             etcdcluster-sample-0
Namespace:        default
Priority:         0
Service Account:  default
Node:             kind-control-plane/172.18.0.2
Start Time:       Tue, 12 Aug 2025 10:40:30 +0530
Labels:           app=etcdcluster-sample
                  apps.kubernetes.io/pod-index=0
                  controller=etcdcluster-sample
                  controller-revision-hash=etcdcluster-sample-754c4cfbcd
                  statefulset.kubernetes.io/pod-name=etcdcluster-sample-0
Annotations:      <none>
Status:           Running
IP:               10.244.0.9
IPs:
  IP:           10.244.0.9
Controlled By:  StatefulSet/etcdcluster-sample
Containers:
  etcd:
    Container ID:  containerd://f74035b8f1ca347d40fdf5ad85fdfd5ebb5ae5d4dce28a80d2553c55d78a52f2
    Image:         gcr.io/etcd-development/etcd:v3.5.21
    Image ID:      gcr.io/etcd-development/etcd@sha256:fd158fbe55240e252947bbd2e8dddc217997ff43978071fac2bd202b6ad15c03
    Ports:         2379/TCP, 2380/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /usr/local/bin/etcd
    Args:
      --name=$(POD_NAME)
      --listen-peer-urls=http://0.0.0.0:2380
      --listen-client-urls=http://0.0.0.0:2379
      --initial-advertise-peer-urls=http://$(POD_NAME).etcdcluster-sample.$(POD_NAMESPACE).svc.cluster.local:2380
      --advertise-client-urls=http://$(POD_NAME).etcdcluster-sample.$(POD_NAMESPACE).svc.cluster.local:2379
    State:          Running
      Started:      Tue, 12 Aug 2025 10:40:36 +0530
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      etcdcluster-sample-state  ConfigMap  Optional: false
    Environment:
      POD_NAME:       etcdcluster-sample-0 (v1:metadata.name)
      POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hgvfp (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  server-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  etcdcluster-sample-server-tls
    Optional:    false
  peer-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  etcdcluster-sample-peer-tls
    Optional:    false
  kube-api-access-hgvfp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  70s   default-scheduler  Successfully assigned default/etcdcluster-sample-0 to kind-control-plane
  Normal  Pulling    70s   kubelet            Pulling image "gcr.io/etcd-development/etcd:v3.5.21"
  Normal  Pulled     64s   kubelet            Successfully pulled image "gcr.io/etcd-development/etcd:v3.5.21" in 5.5s (5.5s including waiting). Image size: 20653564 bytes.
  Normal  Created    64s   kubelet            Created container: etcd
  Normal  Started    64s   kubelet            Started container etcd

@@ -84,6 +91,17 @@ func (r *EtcdClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request)
etcdCluster.Spec.ImageRegistry = r.ImageRegistry
}

// Create Client Certificate for etcd-operator to communicate with the etcdCluster
if etcdCluster.Spec.TLS != nil {
clientCertErr := createClientCertificate(etcdCluster, ctx, r.Client)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comment: usually context is the first parameter for a function or method

Comment on lines +217 to +218
serverCertName := fmt.Sprintf("%s-%s-tls", ec.Name, "server")
peerCertName := fmt.Sprintf("%s-%s-tls", ec.Name, "peer")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please create two functions to generate/get the serverCertName and peerCertName, so that we have a centralized place to maintain the names.

Comment on lines +585 to +588
ExtraConfig: map[string]any{
"issuerName": cmConfig.IssuerName,
"issuerKind": cmConfig.IssuerKind,
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two solutions:

  1. refer to the IssuerNameKey and IssuerNameKey
  2. get the details of populating the ExtraConfig encapsulated in the certificate package.

For simplicity, let's follow solution 1 for now, and revisit 2 in future if needed.

}

func createClientCertificate(ec *ecv1alpha1.EtcdCluster, ctx context.Context, c client.Client) error {
certName := fmt.Sprintf("%s-%s-tls", ec.Name, "client")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create a function to get the secret for client

return ctx
},
)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also verify that we can access the etcdcluster via client certificate.

  • write a key/value pair and read it back

@@ -54,6 +56,11 @@ type EtcdClusterReconciler struct {
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch;get;list;update
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;patch;update;delete
// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick question: I see that we're querying the STS, and checking the podSpec from there, but I'm not sure why we would need to query the pod directly. Am I missing something? Or, is there a particular reason why you added this permission?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Certificate management & usage
4 participants