Internal server error on "ks-controller-manager"

[andrerkn]

Hey guys!
I deployed Kubesphere using last version of CloudFormation template today and now deployed fine!

I am starting to use and I am getting a Internal Server Error when I try to change Admin's default password or when I create a new user for Kubesphere.

Internal error occurred: failed calling webhook "users.iam.kubesphere.io": Post "https://ks-controller-manager.kubesphere-system.svc:443/validate-email-iam-kubesphere-io-v1alpha2?timeout=30s": no endpoints available for service "ks-controller-manager"

$ kubectl get pod --all-namespaces -o wide | grep -i ks-controller-manager

NAMESPACE                      NAME                                               READY   STATUS      RESTARTS         AGE     IP            NODE                          NOMINATED NODE   READINESS GATES
kubesphere-system              ks-controller-manager-64858bb6bf-jpdd4             1/1     Running            9 (7m56s ago)    23m     10.0.15.148   ip-10-0-0-133.ec2.internal    <none>           <none>

$ kubectl describe pod ks-controller-manager-64858bb6bf-jpdd4 -n kubesphere-system
Name:         ks-controller-manager-64858bb6bf-jpdd4
Namespace:    kubesphere-system
Priority:     0
Node:         ip-10-0-0-133.ec2.internal/10.0.0.133
Start Time:   Wed, 13 Apr 2022 17:35:27 +0000
Labels:       app=ks-controller-manager
              pod-template-hash=64858bb6bf
              tier=backend
Annotations:  kubectl.kubernetes.io/restartedAt: 2022-04-13T17:35:20Z
              kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.0.15.148
IPs:
  IP:           10.0.15.148
Controlled By:  ReplicaSet/ks-controller-manager-64858bb6bf
Containers:
  ks-controller-manager:
    Container ID:  docker://2e79bbff9a9924b090cfe4542c1e551c24a05b4443750141aa19bf381cd60f44
    Image:         kubesphere/ks-controller-manager:v3.2.1
    Image ID:      docker-pullable://kubesphere/ks-controller-manager@sha256:db0824cbf2aa611d9be8beede983e7f7412d0a4de291324b1fddb4e7ee00aebc
    Ports:         8080/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      controller-manager
      --logtostderr=true
      --leader-elect=true
    State:          Running
      Started:      Wed, 13 Apr 2022 17:56:21 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 13 Apr 2022 17:51:10 +0000
      Finished:     Wed, 13 Apr 2022 17:51:10 +0000
    Ready:          True
    Restart Count:  9
    Limits:
      cpu:     1
      memory:  1000Mi
    Requests:
      cpu:        30m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /etc/kubesphere/ from kubesphere-config (rw)
      /etc/localtime from host-time (ro)
      /tmp/k8s-webhook-server/serving-certs from webhook-secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-blz7q (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kubesphere-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubesphere-config
    Optional:  false
  webhook-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ks-controller-manager-webhook-cert
    Optional:    false
  host-time:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/localtime
    HostPathType:  
  kube-api-access-blz7q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  27m                   default-scheduler  0/4 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate, 2 Too many pods.
  Normal   Scheduled         27m                   default-scheduler  Successfully assigned kubesphere-system/ks-controller-manager-64858bb6bf-jpdd4 to ip-10-0-0-133.ec2.internal
  Normal   Pulled            27m                   kubelet            Successfully pulled image "kubesphere/ks-controller-manager:v3.2.1" in 108.808398ms
  Normal   Pulled            27m                   kubelet            Successfully pulled image "kubesphere/ks-controller-manager:v3.2.1" in 109.612726ms
  Normal   Pulled            27m                   kubelet            Successfully pulled image "kubesphere/ks-controller-manager:v3.2.1" in 98.921575ms
  Normal   Created           26m (x4 over 27m)     kubelet            Created container ks-controller-manager
  Normal   Started           26m (x4 over 27m)     kubelet            Started container ks-controller-manager
  Normal   Pulled            26m                   kubelet            Successfully pulled image "kubesphere/ks-controller-manager:v3.2.1" in 106.843329ms
  Normal   Pulling           25m (x5 over 27m)     kubelet            Pulling image "kubesphere/ks-controller-manager:v3.2.1"
  Warning  BackOff           7m22s (x94 over 27m)  kubelet            Back-off restarting failed container

Can you help me?

[andrerkn]

I don't know exactly what I did, but it's working now.

Maybe the command below resolved my issue:
kubectl -n kubesphere-system rollout restart deploy ks-controller-manager

Thanks.

[ruiyaoOps]

I'm glad that this problem has been solved.

[FeynmanZhou]

@andrerkn I can't reproduce this issue in my KubeSphere on AWS QS. I could create a new user in KubeSphere.