Starboard operator goes is crash-loop and then it never starts.

[RSE132]

What steps did you take and what happened:

Starboard operator goes is crash-loop and then it never starts. I have deleted the entire deployment and re-deployed it in the k8s cluster...... then it runs properly for few hours and then go into crash-loop state with below errors.

I0103 11:50:14.609297 1 trace.go:205] Trace[776753496]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167 (03-Jan-2022 11:49:14.062) (total time: 60546ms): Trace[776753496]: [1m0.546384684s] [1m0.546384684s] END E0103 11:50:14.609346 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.VulnerabilityReport: failed to list *v1alpha1.VulnerabilityReport: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 189; INTERNAL_ERROR; received from peer {"level":"error","ts":1641210674.061664,"logger":"controller.job","msg":"Could not wait for Cache to sync","reconciler group":"batch","reconciler kind":"Job","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} I0103 11:51:14.062897 1 trace.go:205] Trace[1458497135]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167 (03-Jan-2022 11:50:15.727) (total time: 58335ms): Trace[1458497135]: [58.335395031s] [58.335395031s] END {"level":"error","ts":1641210674.0636806,"logger":"controller.configmap","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ConfigMap","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0764806,"logger":"controller.node","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"Node","error":"failed to wait for node caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0765655,"logger":"controller.configmap","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ConfigMap","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.07677,"logger":"controller.configmap","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ConfigMap","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0768564,"logger":"controller.pod","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"Pod","error":"failed to wait for pod caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.076921,"logger":"controller.cronjob","msg":"Could not wait for Cache to sync","reconciler group":"batch","reconciler kind":"CronJob","error":"failed to wait for cronjob caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0769231,"logger":"controller.daemonset","msg":"Could not wait for Cache to sync","reconciler group":"apps","reconciler kind":"DaemonSet","error":"failed to wait for daemonset caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0766485,"logger":"controller.cronjob","msg":"Could not wait for Cache to sync","reconciler group":"batch","reconciler kind":"CronJob","error":"failed to wait for cronjob caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.076976,"logger":"controller.job","msg":"Could not wait for Cache to sync","reconciler group":"batch","reconciler kind":"Job","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0770347,"logger":"controller.replicaset","msg":"Could not wait for Cache to sync","reconciler group":"apps","reconciler kind":"ReplicaSet","error":"failed to wait for replicaset caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0770917,"logger":"controller.daemonset","msg":"Could not wait for Cache to sync","reconciler group":"apps","reconciler kind":"DaemonSet","error":"failed to wait for daemonset caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0771453,"logger":"controller.pod","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"Pod","error":"failed to wait for pod caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.077109,"logger":"controller.configmap","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ConfigMap","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.077137,"logger":"controller.configmap","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ConfigMap","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0771866,"logger":"controller.replicationcontroller","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ReplicationController","error":"failed to wait for replicationcontroller caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.077208,"msg":"error received after stop sequence was engaged","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.077203,"logger":"controller.job","msg":"Could not wait for Cache to sync","reconciler group":"batch","reconciler kind":"Job","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.077236,"msg":"error received after stop sequence was engaged","error":"failed to wait for node caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0772367,"logger":"controller.replicationcontroller","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ReplicationController","error":"failed to wait for replicationcontroller caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0772595,"msg":"error received after stop sequence was engaged","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0772355,"logger":"controller.job","msg":"Could not wait for Cache to sync","reconciler group":"batch","reconciler kind":"Job","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0772688,"msg":"error received after stop sequence was engaged","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0772898,"logger":"controller.statefulset","msg":"Could not wait for Cache to sync","reconciler group":"apps","reconciler kind":"StatefulSet","error":"failed to wait for statefulset caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.07713,"logger":"controller.configmap","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ConfigMap","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.077299,"logger":"controller.replicaset","msg":"Could not wait for Cache to sync","reconciler group":"apps","reconciler kind":"ReplicaSet","error":"failed to wait for replicaset caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0773025,"msg":"error received after stop sequence was engaged","error":"failed to wait for pod caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0773392,"logger":"controller.configmap","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ConfigMap","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0773573,"msg":"error received after stop sequence was engaged","error":"failed to wait for cronjob caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0775707,"msg":"error received after stop sequence was engaged","error":"failed to wait for daemonset caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.077368,"logger":"controller.configmap","msg":"Could not wait for Cache to sync","reconciler group":"","reconciler kind":"ConfigMap","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.077418,"logger":"controller.statefulset","msg":"Could not wait for Cache to sync","reconciler group":"apps","reconciler kind":"StatefulSet","error":"failed to wait for statefulset caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0773568,"logger":"controller.job","msg":"Could not wait for Cache to sync","reconciler group":"batch","reconciler kind":"Job","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:705"} {"level":"error","ts":1641210674.0776224,"msg":"error received after stop sequence was engaged","error":"failed to wait for cronjob caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0776625,"msg":"error received after stop sequence was engaged","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0776954,"msg":"error received after stop sequence was engaged","error":"failed to wait for replicaset caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0777051,"msg":"error received after stop sequence was engaged","error":"failed to wait for daemonset caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0777304,"msg":"error received after stop sequence was engaged","error":"failed to wait for pod caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0777383,"msg":"error received after stop sequence was engaged","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0777457,"msg":"error received after stop sequence was engaged","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0777593,"msg":"error received after stop sequence was engaged","error":"failed to wait for replicationcontroller caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0777662,"msg":"error received after stop sequence was engaged","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0777838,"msg":"error received after stop sequence was engaged","error":"failed to wait for replicationcontroller caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.077792,"msg":"error received after stop sequence was engaged","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.077799,"msg":"error received after stop sequence was engaged","error":"failed to wait for statefulset caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0778062,"msg":"error received after stop sequence was engaged","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0778396,"msg":"error received after stop sequence was engaged","error":"failed to wait for replicaset caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0778475,"msg":"error received after stop sequence was engaged","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0778542,"msg":"error received after stop sequence was engaged","error":"failed to wait for configmap caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.077861,"msg":"error received after stop sequence was engaged","error":"failed to wait for statefulset caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0778675,"msg":"error received after stop sequence was engaged","error":"failed to wait for job caches to sync: timed out waiting for cache to be synced"} {"level":"error","ts":1641210674.0779657,"logger":"main","msg":"Unable to run starboard operator","error":"starting controllers manager: failed to wait for job caches to sync: timed out waiting for cache to be synced"}

Environment:

• Starboard version (use 11.3):
• Kubernetes version (use kubectl 1.21):

[danielpacak]

From these logs it could be that something is wrong with connection to Kubernetes API server. Could you confirm which K8s platform is it? Upstream Kubernetes or managed cluster such as OpenShift / EKS? Please specify:

• The correct version of Starboard Operator that you deployed (11.3 is not a valid version)
• Installations steps (kubectl or Helm, or OLM)
• Any non-default configuration settings.

For troubleshooting it's useful to know how big is your cluster? How many nodes it has and how many workloads you run. Also could you confirm that the control plane node was READY all them time and the Kubernetes API Server was live and ready?

[RSE132]

k8s platform = AKS, Managed Cluster
version of starboard = starboard-operator:0.13.1
installation steps = kubectl
Non-Default config -
- name: OPERATOR_TARGET_NAMESPACES
value: ""

    resources:
        requests:
          memory: 5Gi
          cpu: 500m
        limits:
          memory: 8Gi
          cpu: 1000m

Cluster Size = 40 nodes

NAME                                STATUS   ROLES   AGE     VERSION
aks-batch-23362585-vmss0005do       Ready    agent   10d     v1.20.7
aks-batch-23362585-vmss0005gg       Ready    agent   3d18h   v1.20.7
aks-batch-23362585-vmss0005h8       Ready    agent   38h     v1.20.7
aks-batch-23362585-vmss0005hv       Ready    agent   14h     v1.20.7
aks-batch-23362585-vmss0005hx       Ready    agent   10h     v1.20.7
aks-batch-23362585-vmss0005i5       Ready    agent   52m     v1.20.7
aks-batch-23362585-vmss0005i6       Ready    agent   20m     v1.20.7
aks-batch-23362585-vmss0005i7       Ready    agent   9m8s    v1.20.7
aks-default-23362585-vmss0000ch     Ready    agent   35d     v1.20.7
aks-default-23362585-vmss0000ci     Ready    agent   35d     v1.20.7
aks-default-23362585-vmss0000cn     Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000tl       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000tp       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000uc       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000ue       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000uf       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000ug       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000uo       Ready    agent   36d     v1.20.7
aks-large-23362585-vmss0000us       Ready    agent   36d     v1.20.7
aks-large-23362585-vmss0000ut       Ready    agent   36d     v1.20.7
aks-large-23362585-vmss0000uy       Ready    agent   36d     v1.20.7
aks-large-23362585-vmss0000v0       Ready    agent   36d     v1.20.7
aks-large-23362585-vmss0000v2       Ready    agent   36d     v1.20.7
aks-large-23362585-vmss0000v6       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000v8       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000vi       Ready    agent   35d     v1.20.7
aks-large-23362585-vmss0000vn       Ready    agent   33d     v1.20.7
aks-large-23362585-vmss0000vo       Ready    agent   33d     v1.20.7
aks-large-23362585-vmss0000vs       Ready    agent   31d     v1.20.7
aks-large-23362585-vmss0000vu       Ready    agent   22d     v1.20.7
aks-large-23362585-vmss0000w2       Ready    agent   20d     v1.20.7
aks-large-23362585-vmss0000wc       Ready    agent   14d     v1.20.7
aks-large-23362585-vmss0000wi       Ready    agent   14d     v1.20.7
aks-large-23362585-vmss0000wm       Ready    agent   14d     v1.20.7
aks-large-23362585-vmss0000wp       Ready    agent   13d     v1.20.7
aks-large-23362585-vmss0000wq       Ready    agent   13d     v1.20.7
aks-large-23362585-vmss0000ws       Ready    agent   13d     v1.20.7
aks-large-23362585-vmss0000wv       Ready    agent   13d     v1.20.7
aks-large-23362585-vmss0000ww       Ready    agent   13d     v1.20.7
aks-large-23362585-vmss0000wy       Ready    agent   9d      v1.20.7
aks-sparkspot-23362585-vmss0001k4   Ready    agent   9h      v1.20.7

workload => RS=1009 SS=157 DS=7 & Jobs=3643

Control Plane Status = Live
KUBE API = Live

[danielpacak]

Thank you for providing additional details @RSE132. It's still hard to figure out the root cause of the problems you are facing. Also, we have a limited capacity and permissions to access your environment and troubleshoot managed K8s clusters (possibly configured with a managed registry).

Therefore, I can only ask the community members familiar with AKS for help or point you to the contributing guide where you can find some hints on debugging the operator by running it Out of Cluster.

[RSE132]

@danielpacak Not sure why... but this issue I am facing on a specific cluster only

The error starts with -
E0127 15:37:13.497966 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.VulnerabilityReport: failed to list *v1alpha1.VulnerabilityReport: the server was unable to return a response in the time allotted, but may still be processing the request (get vulnerabilityreports.aquasecurity.github.io)

Looks like it fails to list v1alpha1.VulnerabilityReport as it is timing out.... Can we increase this time out value ?

[danielpacak]

We are using https://github.com/kubernetes-sigs/controller-runtime, which instantiates the default client.Client to communicate with K8s API server. I believe it's configurable, but currently we don't expose such config as Starboard settings.

[RSE132]

This would be good to have a new feature to configure these settings.... In my situation there are 5000+ reports and to get all these api call times out which is set to 60s by default.... allowing this timeout value configurable would resolve such issue in bigger cluster....

[cdesaintleger]

Hello @RSE132 , did you find a temporary solution ?
we have the same issue on a cluster with 8000+ pods.