Kong Performance Problem on EKS

[kartef]

Issue Summary: I'm experiencing unusually long response times when querying an upstream service (host.example.com) through a Kong ingress controller running in an Amazon EKS (Elastic Kubernetes Service) cluster:

curl -o /dev/null -s -w "DNS Lookup: %{time_namelookup}s\nTCP Connect: %{time_connect}s\nServer Response: %{time_starttransfer}s\nTotal Time: %{time_total}s\n" -v host.example.com

• Trying x.x.x.x:80...
• Connected tohost.example.com (x.x.x.x) port 80 (#0)

GET / HTTP/1.1
Host: host.example.com
User-Agent: curl/7.76.1
Accept: /

• Mark bundle as not supporting multiuse
  < HTTP/1.1 307 Temporary Redirect
  < Content-Length: 0
  < Connection: keep-alive
  < location: /ui
  < date: Wed, 30 Oct 2024 18:20:06 GMT
  < X-Kong-Upstream-Latency: 2
  < X-Kong-Proxy-Latency: 1
  < Via: kong/3.6.1
  < X-Kong-Request-Id: eb1872148f56f86e1de50f6563fd1703
  <
• Connection #0 to host host.example.com left intact
  DNS Lookup: 0.005319s
  TCP Connect: 0.027189s
  Server Response: 19.489237s
  Total Time: 19.489647s

I'm attempting to analyze the network path and identify any bottlenecks or latency sources. Here’s the setup and what I’ve tried so far:

Setup:

The Kong ingress is running in an EKS cluster on EC2 instances, and the upstream service (test-ui) is also within the same VPC.
I've confirmed that both curl and nslookup can successfully resolve and connect to host.example.com, but the response times are high.
Example response time metrics using curl on this endpoint frequently show significant delays in the time_connect and time_starttransfer phases.
Troubleshooting Steps Taken:

Initial tcpdump Attempts:

I added a sidecar container with tcpdump to the Kong pods to capture network traffic: https://support.konghq.com/support/s/article/How-can-I-take-a-packet-capture-in-a-kong-pod-in-a-kubernetes-environment

However, when running tcpdump commands like:

tcpdump -npi any -As0 -w /tmp/packet.pcap host host.example.com and port 80

Same Problems on Port 443

I consistently see 0 packets captured despite actively sending traffic to the endpoint via curl.
The packets are received by the filter, but no capture data is saved.
Network and DNS Verifications:

I verified that DNS resolution works
Security groups and NACLs have been checked, with ICMP and necessary TCP ports open.

Questions:

Has anyone encountered tcpdump capturing 0 packets while filters detect traffic? I would appreciate insights on possible causes or misconfigurations within an EKS/Kong setup.
Are there alternative ways to trace and troubleshoot latency issues in Kubernetes/EKS environments, especially for ingress-to-upstream traffic?

[randmonkey]

From X-Kong-Upstream-Latency and X-Kong-Proxy-Latency headers, we can see that the duration from Kong receiving the request to Kong sending the response is 2+1=3ms:
https://docs.konghq.com/gateway/latest/reference/configuration/#headers
So the major latency happens between your client and Kong gateway.
KIC does not handle traffic, it only generates Kong configuration from your k8s resources (Ingess,Service and so on). All traffic go to Kong gateway.