-
Notifications
You must be signed in to change notification settings - Fork 640
-
Notifications
You must be signed in to change notification settings - Fork 640
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gRPC-JS client hangs when target server restarts #2785
Comments
It looks like you enabled only connection management trace logs, but we will also need trace logs for individual requests to determine what is happening with requests that is causing them not to be sent. It's probably simplest to just use |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem description
gRPC js client freezes on server disconnections, require full client restarts
Reproduction steps
We have a server (server API is gRPC, using malijs), let's call it A which calls a gRPC target proxied by an NLB and nginx gRPC ingress, let's call it B. If we go ahead and restart the target instances of B, The client might sometime hang, resulting in the application A failing to function. All flows calling to A that will result in A calling B will hang
The moment we restart A, it becomes functioning, meaning it's a client state issue.
Environment
{ 'grpc.keepalive_time_ms': 10_000, 'grpc.max_receive_message_length': 64 * 1024 * 1024 }
Additional context
We've activated gRPC tracing on server A
here's what a normal server selection trace looks like:
As expected, connections enter into
TRANSIENT_FAILURE
state once I kill the target pods B. When they become responsive again, We get these messages:And from that moment on, we stop seeing these messages altogether. As though the driver becomes stuck.
We still see these traces
but no request is actuall being sent, the client is completely frozen. Any help will be much obliged
The text was updated successfully, but these errors were encountered: