[BUG]: Data race in gRPC client interceptor when using grpc.Peer() CallOption

[lihao-figma]

Tracer Version(s)

1.65.1

Go Version(s)

1.22

Bug Report

gRPC version: v1.73.0

  Race Detector Output

  WARNING: DATA RACE
  Read at 0x00c0007e9920 by goroutine 380:
    gopkg.in/DataDog/dd-trace-go.v1/contrib/google.golang.org/grpc.doClientRequest()
        contrib/google.golang.org/grpc/client.go:197 +0x506

  Previous write at 0x00c0007e9920 by goroutine 763:
    google.golang.org/grpc.PeerCallOption.after()
        google.golang.org/grpc@v1.73.0/rpc_util.go:271 +0xbe

This happens upon stream grpc call with fast context cancellation after start.

The issue occurs in doClientRequest (client.go):

  // line 191-197
  var p peer.Peer
  opts = append(opts, grpc.Peer(&p))

  handlerCtx := injectSpanIntoContext(ctx)
  err := handler(handlerCtx, opts)

  setSpanTargetFromPeer(span, p)  // RACE: Reading p here

in grpc library
google.golang.org/grpc@v1.73.0/stream.go line 391

	if desc != unaryStreamDesc {
		// Listen on cc and stream contexts to cleanup when the user closes the
		// ClientConn or cancels the stream context.  In all other cases, an error
		// should already be injected into the recv buffer by the transport, which
		// the client will eventually receive, and then we will cancel the stream's
		// context in clientStream.finish.
		go func() {
			select {
			case <-cc.ctx.Done():
				cs.finish(ErrClientConnClosing)
			case <-ctx.Done():
				cs.finish(toRPCErr(ctx.Err()))
			}
		}()
	}

The race happens because:

1. A local peer.Peer variable is created and passed to gRPC via grpc.Peer(&p)
2. The handler (which creates the gRPC stream/call) returns immediately
3. DataDog immediately reads from p via setSpanTargetFromPeer()
4. Meanwhile, gRPC populates the peer asynchronously (if context cancellation signal received) when the call completes, calling PeerCallOption.after() which writes to p

The grpc.Peer() CallOption is designed to be populated after the RPC completes, not immediately after stream creation. The peer information is only available once the connection is established and the call finishes. DataDog's code incorrectly assumes the peer is populated synchronously when the handler returns.

We can consider change fetching peer information from stream context?

  if p, ok := peer.FromContext(stream.Context()); ok {
      setSpanTargetFromPeer(span, *p)
  }

[rarguelloF]

Hello @lihao-figma and thanks for the detailed report.

I've tried to reproduce without success. Would you mind providing a minimal example that would reproduce the issue?

For reference, this is what I tried (attempted to reproduce the conditions you described, please let me know if something looks wrong):

Inside contrib/google.golang.org/grpc/grpc_test.go

func TestPeerRace(t *testing.T) {
	mt := mocktracer.Start()
	defer mt.Stop()

	rig, err := newRig(true)
	require.NoError(t, err)

	// not using rig.Close() here because GracefulStop waits for pending calls to finish
	// and it takes a long time since we are calling the slow endpoint
	defer func() {
		rig.server.Stop()
		rig.conn.Close()
	}()

	// Run many concurrent calls with the slow endpoint to increase likelihood of hitting the race
	const numGoroutines = 50
	const numIterations = 20

	done := make(chan bool)

	for g := 0; g < numGoroutines; g++ {
		go func() {
			for i := 0; i < numIterations; i++ {
				// Create a context with a short timeout - the slow endpoint takes 5s to finish
				// but we cancel after 100ms.
				ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)

				// Start a streaming call with the slow endpoint
				stream, err := rig.client.StreamPing(ctx)
				require.NoError(t, err)

				_ = stream.Send(&fixturepb.FixtureRequest{Name: "slow"})
				_ = stream.CloseSend()
				cancel()
			}
			done <- true
		}()
	}

	for g := 0; g < numGoroutines; g++ {
		<-done
	}
}

❯ go test -race -count 100 -run TestPeerRace ./contrib/google.golang.org/grpc -v
[...]
=== RUN   TestPeerRace
--- PASS: TestPeerRace (0.19s)
PASS
ok  	github.com/DataDog/dd-trace-go/contrib/google.golang.org/grpc/v2	12.619s