Support / clarify/ feature request for gRPC→gRPC traffic mirroring using mirror: true in HTTPProxy

[roshan-meesho]

Please describe the problem you have
We have a gRPC service (Go-based) running behind Contour using HTTPProxy with protocol: h2c serving grpc traffic.
We are trying to mirror this traffic to another gRPC backend (written in Rust in another VM) using the HTTPProxy mirror: true option.

Example:
        - name: xxxx
          port: 80
          slowStartPolicy:
            aggression: '0.1'
            minWeightPercent: 10
            window: 300s
        - name: xxx-mirror-rust
          mirror: true
          port: 80
          protocol: h2c
          weight: 1

and we have pointed that service xxx-mirror-rust to a externalname which is poiting to my hosted rust backend at a certain port.

What did we observe?

• Mirroring works fine for HTTP applications serving http traffic.
• For gRPC traffic, we consistently see 503 errors on the mirror service.
• It seems gRPC → gRPC mirroring might not be supported yet or documented.

What do we expect?

• Clarification on whether Contour supports gRPC mirroring on grpc traffic with mirror:true option as working for http traffic.
• If not, request to add support or documentation for this use case.
• Any known workarounds for sending gRPC mirror traffic to another gRPC backend.

Used Environment

• Contour version: 1.27.0 & 1.28.0
• Used HTTPProxy with protocol: h2c for both upstreams as grpc needs this annotations

[github-actions]

Hey @roshan-meesho! Thanks for opening your first issue. We appreciate your contribution and welcome you to our community! We are glad to have you here and to have your input on Contour. You can also join us on our mailing list and in our channel in the Kubernetes Slack Workspace

[tsaarni]

There shouldn't be anything special in mirroring gRPC as far as I know but since you use Service of type: ExternalName you need to enable that separately in Contour configuration by setting enableExternalNameService: true. See here https://projectcontour.io/docs/1.33/config/external-service-routing/ and see also security advisory on using external name services GHSA-5ph6-qq5x-7jwc.

[tsaarni]

For gRPC traffic, we consistently see 503 errors on the mirror service.

One more thing: I’m not sure what you mean by seeing 503 errors on the mirror service. Who is actually returning those 503 responses, and who is receiving them? If the mirror service itself returns error when processing the request, then it’s a different issue.

[roshan-meesho]

Its working now

[roshan-meesho]

hey @tsaarni, things got worked but its not reusing the connections instead its making new connections due to which main service is getting throttled. Is there any work around here on how mirroring can reuse the connections.

CPU and memory are fine, the issue isn’t resource load but the number of connections
During mirroring, Envoy opens a new TCP connection for every request instead of reusing existing ones.
At high RPS, that floods the OS with short-lived sockets, hitting port and FD limits, so new connections fail and give 504 gateway timeouts even though the VM looks idle.

When I reused connections and hit the service from local, throughput stabilized at 500~1000 RPS so most probablt it is a connection reuse issue, not a VM limit.

You could see the new connections inc from this service which is mirrored This image shows that it fails establishing connections the mirroring requests(new ones) which are getting created

[siddharth-pal-meesho]

@tsaarni can you pls check this? does contour supports reuse of connection in mirrored traffic?

[tsaarni]

I captured the traffic with Wireshark and confirmed that Envoy reuses connections to the mirror service in the same way as it does for the primary service. Envoy maintains multiple keep-alive connections rather than just one.

I don’t recall or even know the exact details how it decides on the number of connections to set up, but I think the number of connections was related to the number of Envoy worker threads, which in turn is related to the number of CPU cores on the host running Envoy. In my laptop with 20 CPU threads, I can see around 20 connections from Envoy to the upstream mirror service.

[github-actions]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack