Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky Test: xds/TestUnmarshalListener_WithUpdateValidatorFunc #7599

Open
arjan-bal opened this issue Sep 9, 2024 · 2 comments
Open

Flaky Test: xds/TestUnmarshalListener_WithUpdateValidatorFunc #7599

arjan-bal opened this issue Sep 9, 2024 · 2 comments
Assignees
Labels
Area: Testing Includes tests and testing utilities that we have for unit and e2e tests within our repo. Type: Bug Type: Testing

Comments

@arjan-bal
Copy link
Contributor

Sample run: https://github.com/grpc/grpc-go/actions/runs/10779216338/job/29892227988?pr=7498

Error message:

xds_server_integration_test.go:115: Serve() failed: grpc: the server has been stopped

job-logs.txt

@arjan-bal arjan-bal added Type: Bug Type: Testing Area: Testing Includes tests and testing utilities that we have for unit and e2e tests within our repo. labels Sep 9, 2024
@arjan-bal
Copy link
Contributor Author

@easwars
Copy link
Contributor

easwars commented Sep 10, 2024

I think this is what is happening:

  • The test starts an xds-enabled gRPC server here:
    lis, cleanup2 := setupGRPCServer(t, bootstrapContents)
  • setupGRPCServer creates an xds-enabled gRPC server and spawns a goroutine to start serving here:
  • Back in the main test goroutine though, for sub-tests where we expect failures, we create a really short deadline and attempt to make an RPC and expect it to fail.
    // Make a context with a shorter timeout from the top level test
  • What can happen here is that the RPC can fail before the server has stared serving, and the test completes.
  • A little later on though, the goroutine to start serving runs, and sees that the server has already been closed.

We need to ensure that the server has started serving before attempting client RPCs.

@arjan-bal arjan-bal self-assigned this Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Testing Includes tests and testing utilities that we have for unit and e2e tests within our repo. Type: Bug Type: Testing
Projects
None yet
Development

No branches or pull requests

2 participants