KIC pod fails to detect new kong-admin IP, when new pod comes up intermittently.

[pratham-pay]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

We are getting below error most of the times in KIC Pod for newly provision kong-admin pod

Error: unable to build kong api client(s): client not ready: making HTTP request:
Get "https://x.x.x.x:8444/status": dial tcp x.x.x.x:8444: connect: connection refused

this is happening intermittently.

Expected Behavior

ideally kong KIC should notice the newly created kong-admin pod and only send request to new IP.
kong kic is discovering kong-admin through k8s service.

service/kong-prod-kong-admin                    NodePort       172.20.121.91    <none>                                                 8001:32256/TCP,8444:31192/TCP   297d
endpoints/kong-prod-kong-admin                 x.x.x.x:8001,x.x.x.x:8001,x.x.x.x:8444 ,x.x.x.x:8444   297d

But somehow kong-kic is caching the old ip detail. and tries to connect to it.This is impacting the new changes that we are adding in the kong gateway.

Steps To Reproduce

This is happening intermittently.

Kong Ingress Controller version

image: kong/kubernetes-ingress-controller:3.1

Kubernetes version

EKS version 1.29

Anything else?

No response