Rancher not works right with Higress

[jaggerwang]

Ⅰ. Issue Description

When run kubectl get pods, got some error.

shell@Alicloud:~$ kubectl get pods -n cattle-system
E0601 10:33:54.512853    1624 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:33:56.738290    1624 memcache.go:255] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:33:56.796198    1624 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:33:56.877312    1624 memcache.go:106] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:33:57.457092    1624 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:33:57.500064    1624 memcache.go:106] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:33:57.654873    1624 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:33:57.706508    1624 memcache.go:106] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
NAME                                    READY   STATUS    RESTARTS   AGE
cattle-cluster-agent-854869c948-lwzxc   0/1     Pending   0          7m6s
shell@Alicloud:~$ kubectl get pods -n basicai-backend
E0601 10:34:21.473206    1692 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:34:24.876186    1692 memcache.go:255] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:34:24.936207    1692 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:34:25.201906    1692 memcache.go:106] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:34:25.871095    1692 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:34:25.950778    1692 memcache.go:106] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:34:26.861167    1692 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0601 10:34:26.912481    1692 memcache.go:106] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
NAME                                                  READY   STATUS    RESTARTS   AGE
admin-648df54b-hzbnv                                  0/1     Pending   0          27m
admin-648df54b-sq648                                  0/1     Pending   0          27m

Ⅱ. Describe what happened

I want to try Higress in my alidev ACK cluster, this cluster is managed by Rancher. After I installed higress, following the instruction https://higress.io/zh-cn/docs/user/quickstart, the rancher cluster agent running in ACK crashed again and again, and the cluster is disconnected from Rancher, but I can access the Higress's web console. So I delete Higress, but the cluster is still cannot import to Rancher and not works rightly.

Ⅵ. Environment:

• Higress version: 1.0
• Kubernetes version : Aliyun ACK 1.22.15-aliyun.1
• Others:

[johnlanni]

the rancher cluster agent running in ACK crashed again and again

I think there is a problem with cattle-cluster-agent, you need to execute kubectl logs -p to see the previous container's logs, then you will know what happened (probably rancher's crash error).

[johnlanni]

@jaggerwang here

[jaggerwang]

Rancher agent log

time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=Token"
time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for flowcontrol.apiserver.k8s.io/v1beta1, Kind=PriorityLevelConfiguration"
time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for storage.k8s.io/v1beta1, Kind=CSIStorageCapacity"
time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for bookkeeper.streamnative.io/v1alpha1, Kind=BookKeeperCluster"
time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=AuthConfig"
time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for management.cattle.io/v3, Kind=Group"
time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole"
time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for pingcap.com/v1alpha1, Kind=DMCluster"
time="2023-06-01T09:08:24Z" level=info msg="Watching metadata for networking.k8s.io/v1, Kind=IngressClass"
W0601 09:08:24.894289      39 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0601 09:08:25.270026      39 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
I0601 09:09:11.278977      39 leaderelection.go:258] successfully acquired lease kube-system/cattle-controllers
time="2023-06-01T09:09:12Z" level=info msg="Steve auth startup complete"
time="2023-06-01T09:09:12Z" level=info msg="Registering namespaceHandler for adding labels "
W0601 09:09:13.204380      39 warnings.go:80] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
W0601 09:09:13.280049      39 warnings.go:80] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
time="2023-06-01T09:09:13Z" level=info msg="Starting catalog.cattle.io/v1, Kind=Operation controller"
time="2023-06-01T09:09:13Z" level=info msg="Starting admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration controller"
time="2023-06-01T09:09:13Z" level=info msg="Starting apps/v1, Kind=Deployment controller"
E0601 09:09:13.447021      39 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 12171 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3ccd600, 0x6eb69c0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x86
panic(0x3ccd600, 0x6eb69c0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/rancher/rancher/pkg/controllers/managementagent/endpoints.convertIngressToServicePublicEndpointsMap(0x4d318f0, 0xc016458300, 0x0, 0xc000074026, 0x5, 0x1)
	/go/src/github.com/rancher/rancher/pkg/controllers/managementagent/endpoints/endpoints.go:312 +0x6cb
github.com/rancher/rancher/pkg/controllers/managementagent/endpoints.(*WorkloadEndpointsController).UpdateEndpoints(0xc0104cbe40, 0xc00c8d16e0, 0x1e, 0xc0164761e0, 0x3a919a0, 0xc00fd836e0)
	/go/src/github.com/rancher/rancher/pkg/controllers/managementagent/endpoints/workload_endpoints.go:89 +0x674
github.com/rancher/rancher/pkg/controllers/managementagent/workload.(*CommonController).syncDeployments(0xc000156d20, 0xc00c8d16e0, 0x1e, 0xc013151af8, 0x10, 0x7fc9d91c7108, 0x10, 0xc00fd836b0)
	/go/src/github.com/rancher/rancher/pkg/controllers/managementagent/workload/workload_common.go:135 +0x1a4
github.com/rancher/rancher/pkg/generated/norman/apps/v1.(*deploymentController).AddHandler.func1(0xc00c8d16e0, 0x1e, 0x448c440, 0xc013151af8, 0xc01226bb58, 0x3, 0x3, 0x0)
	/go/src/github.com/rancher/rancher/pkg/generated/norman/apps/v1/zz_generated_deployment_controller.go:156 +0x67
github.com/rancher/norman/controller.(*genericController).AddHandler.func1(0xc00c8d16e0, 0x1e, 0x4c80c98, 0xc013151af8, 0xc013151af8, 0x4d2e9d8, 0xc013151af8, 0x1)
	/go/pkg/mod/github.com/rancher/[email protected]/controller/generic_controller.go:62 +0x224
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc0104dbc80, 0xc00c8d16e0, 0x1e, 0x4c80c98, 0xc013151af8, 0x0, 0xc013151af8, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/sharedcontroller.go:29 +0x4e
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc00b044200, 0xc00c8d16e0, 0x1e, 0x4c80c98, 0xc013151af8, 0xc01623eb01, 0x0)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/sharedhandler.go:69 +0x14c
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc0008a96b0, 0xc00c8d16e0, 0x1e, 0xc00e35ace8, 0x0)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:215 +0xd1
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc0008a96b0, 0x3a919a0, 0xc00ec0fef0, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:197 +0xe7
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc0008a96b0, 0x203001)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:174 +0x54
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:163
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0140dc490)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0140dc490, 0x4c53380, 0xc016236570, 0x1, 0xc00e015f80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0140dc490, 0x3b9aca00, 0x0, 0x1, 0xc00e015f80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc0140dc490, 0x3b9aca00, 0xc00e015f80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by github.com/rancher/lasso/pkg/controller.(*controller).run
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:134 +0x33b
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2f85a2b]

goroutine 12171 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x109
panic(0x3ccd600, 0x6eb69c0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/rancher/rancher/pkg/controllers/managementagent/endpoints.convertIngressToServicePublicEndpointsMap(0x4d318f0, 0xc016458300, 0x0, 0xc000074026, 0x5, 0x1)
	/go/src/github.com/rancher/rancher/pkg/controllers/managementagent/endpoints/endpoints.go:312 +0x6cb
github.com/rancher/rancher/pkg/controllers/managementagent/endpoints.(*WorkloadEndpointsController).UpdateEndpoints(0xc0104cbe40, 0xc00c8d16e0, 0x1e, 0xc0164761e0, 0x3a919a0, 0xc00fd836e0)
	/go/src/github.com/rancher/rancher/pkg/controllers/managementagent/endpoints/workload_endpoints.go:89 +0x674
github.com/rancher/rancher/pkg/controllers/managementagent/workload.(*CommonController).syncDeployments(0xc000156d20, 0xc00c8d16e0, 0x1e, 0xc013151af8, 0x10, 0x7fc9d91c7108, 0x10, 0xc00fd836b0)
	/go/src/github.com/rancher/rancher/pkg/controllers/managementagent/workload/workload_common.go:135 +0x1a4
github.com/rancher/rancher/pkg/generated/norman/apps/v1.(*deploymentController).AddHandler.func1(0xc00c8d16e0, 0x1e, 0x448c440, 0xc013151af8, 0xc01226bb58, 0x3, 0x3, 0x0)
	/go/src/github.com/rancher/rancher/pkg/generated/norman/apps/v1/zz_generated_deployment_controller.go:156 +0x67
github.com/rancher/norman/controller.(*genericController).AddHandler.func1(0xc00c8d16e0, 0x1e, 0x4c80c98, 0xc013151af8, 0xc013151af8, 0x4d2e9d8, 0xc013151af8, 0x1)
	/go/pkg/mod/github.com/rancher/[email protected]/controller/generic_controller.go:62 +0x224
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc0104dbc80, 0xc00c8d16e0, 0x1e, 0x4c80c98, 0xc013151af8, 0x0, 0xc013151af8, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/sharedcontroller.go:29 +0x4e
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc00b044200, 0xc00c8d16e0, 0x1e, 0x4c80c98, 0xc013151af8, 0xc01623eb01, 0x0)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/sharedhandler.go:69 +0x14c
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc0008a96b0, 0xc00c8d16e0, 0x1e, 0xc00e35ace8, 0x0)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:215 +0xd1
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc0008a96b0, 0x3a919a0, 0xc00ec0fef0, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:197 +0xe7
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc0008a96b0, 0x203001)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:174 +0x54
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:163
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0140dc490)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0140dc490, 0x4c53380, 0xc016236570, 0x1, 0xc00e015f80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0140dc490, 0x3b9aca00, 0x0, 0x1, 0xc00e015f80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc0140dc490, 0x3b9aca00, 0xc00e015f80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by github.com/rancher/lasso/pkg/controller.(*controller).run
	/go/pkg/mod/github.com/rancher/[email protected]/pkg/controller/controller.go:134 +0x33b

[johnlanni]

https://github.com/rancher/rancher/blob/release/v2.7/pkg/controllers/managementagent/endpoints/endpoints.go#L312
This is a rancher bug, which suppose backend.service must have a value, without considering the resource backend.

https://kubernetes.io/docs/concepts/services-networking/ingress/#resource-backend