ccl/multiregionccl: TestRegionLivenessProberForLeases failed

[cockroach-teamcity]

ccl/multiregionccl.TestRegionLivenessProberForLeases failed with artifacts on master @ 0d284cc52708504d8c71c4358041f45fcf792a52:

Fatal error:

panic: test timed out after 15m0s
running tests:
	TestRegionLivenessProberForLeases (12m13s)

Stack:

goroutine 458684 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38

Log preceding fatal error

* github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_BatchStream_Handler({0x6932f40?, 0x401b247508}, {0x83464e8, 0x402705b0e0})
* 	bazel-out/aarch64-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10718 +0xdc
* github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x6932f40, 0x401b247508}, {0x83464e8, 0x402705b0e0}, 0x40430ac240, 0x6e5ab50)
* 	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:143 +0x374
* google.golang.org/grpc.getChainStreamHandler.func1({0x6932f40, 0x401b247508}, {0x83464e8, 0x402705b0e0})
* 	external/org_golang_google_grpc/server.go:1504 +0xa0
* github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x6932f40, 0x401b247508}, {0x83464e8, 0x402705b0e0}, 0x40430ac240?, 0x4026decc80)
* 	pkg/rpc/context.go:175 +0x78
* google.golang.org/grpc.getChainStreamHandler.func1({0x6932f40, 0x401b247508}, {0x83464e8, 0x402705b0e0})
* 	external/org_golang_google_grpc/server.go:1504 +0xa0
* github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0x40032ca000?, {{0x6527440?}, {0x8367ee0?, 0x40157ce300?}}}, {0x6932f40, 0x401b247508}, {0x83464e8, 0x402705b0e0}, 0x40430ac240, 0x4026decc40)
* 	pkg/rpc/auth.go:152 +0x2e4
* google.golang.org/grpc.getChainStreamHandler.func1({0x6932f40, 0x401b247508}, {0x83464e8, 0x402705b0e0})
* 	external/org_golang_google_grpc/server.go:1504 +0xa0
* github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x4013e9d9c8?, 0x8309fb0?})
* 	pkg/rpc/context.go:142 +0x34
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x40470aa630, {0x8309fb0, 0x406740bef0}, {0x401e2c5890, 0x27}, 0x4013e9da88)
* 	pkg/util/stop/stopper.go:350 +0x100
* github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x6932f40, 0x401b247508}, {0x83464e8, 0x402705b0e0}, 0x40430ac240, 0x4026decc00)
* 	pkg/rpc/context.go:141 +0xc0
* google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x6932f40, 0x401b247508}, {0x83464e8, 0x402705b0e0}, 0x40430ac240, 0x402ec12920?)
* 	external/org_golang_google_grpc/server.go:1495 +0x88
* google.golang.org/grpc.(*Server).processStreamingRPC(0x4009a970e0, {0x8369c80, 0x402bb01d40}, 0x402703dc20, 0x40139c2c60, 0xc40dfa0, 0x0)
* 	external/org_golang_google_grpc/server.go:1659 +0xf78
* google.golang.org/grpc.(*Server).handleStream(0x4009a970e0, {0x8369c80, 0x402bb01d40}, 0x402703dc20, 0x0)
* 	external/org_golang_google_grpc/server.go:1739 +0x780
* google.golang.org/grpc.(*Server).serveStreams.func1.1()
* 	external/org_golang_google_grpc/server.go:970 +0xb4
* created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 276263
* 	external/org_golang_google_grpc/server.go:981 +0x14c
* 
* goroutine 265887 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	pkg/util/admission/work_queue.go:424 +0x7c
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 265279
* 	pkg/util/admission/work_queue.go:421 +0x2ec
* 
* goroutine 265936 [chan receive, 9 minutes]:
* github.com/cockroachdb/pebble.(*fileCacheShard).releaseLoop(0x4003a16630)
* 	external/com_github_cockroachdb_pebble/file_cache.go:397 +0x80
* created by github.com/cockroachdb/pebble.(*fileCacheShard).init in goroutine 265279
* 	external/com_github_cockroachdb_pebble/file_cache.go:388 +0xe4
* 
* goroutine 265892 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	pkg/util/admission/work_queue.go:501 +0x140
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 265279
* 	pkg/util/admission/work_queue.go:472 +0x5c
* 
*

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-47353