kv/kvserver: TestLogGrowthWhenRefreshingPendingCommands failed

[cockroach-teamcity]

kv/kvserver.TestLogGrowthWhenRefreshingPendingCommands failed on release-24.3 @ e4dfca280a15812780ddf6d494899a9b2b61f5c8:

          | runtime.goexit
          | 	src/runtime/asm_amd64.s:1695
        Wraps: (2) "a": transfer lease unexpected error
        Wraps: (3) forced error mark
          | "originated at breaker breaker"
          | github.com/cockroachdb/cockroach/pkg/util/circuit/*circuit.breakerErrorMark::
        Wraps: (4) forced error mark
          | "breaker open"
          | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        Wraps: (5) replica unavailable: (n1,s1):1 unable to serve request to r70:{a-/Table/0} [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n4,s4):4, (n5,s5):5, next=6, gen=7, sticky=9223372036.854775807,2147483647]: closed timestamp: 1735645098.596740593,0 (2024-12-31 11:38:18); raft status: {"id":"1","term":6,"vote":"1","commit":7053,"lead":"1","leadEpoch":"0","raftState":"StateLeader","applied":7053,"progress":{"5":{"match":5321,"next":5386,"state":"StateReplicate"},"1":{"match":7377,"next":7378,"state":"StateReplicate"},"2":{"match":7377,"next":7378,"state":"StateReplicate"},"3":{"match":7053,"next":7054,"state":"StateProbe"},"4":{"match":5501,"next":5566,"state":"StateReplicate"}},"leadtransferee":"0"}: have been waiting 60.01s for slow proposal RequestLease ["a"]
        Wraps: (6)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaUnavailableError
          | 	pkg/kv/kvserver/replica_circuit_breaker.go:268
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).replicaUnavailableError
          | 	pkg/kv/kvserver/replica_circuit_breaker.go:277
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaCircuitBreaker).tripSync
          | 	pkg/kv/kvserver/replica_circuit_breaker.go:97
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaCircuitBreaker).TripAsync.func1
          | 	pkg/kv/kvserver/replica_circuit_breaker.go:91
          | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
          | 	pkg/util/stop/stopper.go:498
          | runtime.goexit
          | 	src/runtime/asm_amd64.s:1695
        Wraps: (7) closed timestamp: 1735645098.596740593,0 (2024-12-31 11:38:18); raft status: {"id":"1","term":6,"vote":"1","commit":7053,"lead":"1","leadEpoch":"0","raftState":"StateLeader","applied":7053,"progress":{"5":{"match":5321,"next":5386,"state":"StateReplicate"},"1":{"match":7377,"next":7378,"state":"StateReplicate"},"2":{"match":7377,"next":7378,"state":"StateReplicate"},"3":{"match":7053,"next":7054,"state":"StateProbe"},"4":{"match":5501,"next":5566,"state":"StateReplicate"}},"leadtransferee":"0"}
        Wraps: (8)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).refreshProposalsLocked
          | 	pkg/kv/kvserver/replica_raft.go:1726
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
          | 	pkg/kv/kvserver/replica_raft.go:1558
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
          | 	pkg/kv/kvserver/store_raft.go:711
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
          | 	pkg/kv/kvserver/scheduler.go:410
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
          | 	pkg/kv/kvserver/scheduler.go:319
          | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
          | 	pkg/util/stop/stopper.go:498
          | runtime.goexit
          | 	src/runtime/asm_amd64.s:1695
        Wraps: (9) have been waiting 60.01s for slow proposal RequestLease ["a"]
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *markers.withMark (4) *markers.withMark (5) *kvpb.ReplicaUnavailableError (6) *errbase.opaqueWrapper (7) *errutil.withPrefix (8) *errbase.opaqueWrapper (9) *errutil.leafError
    --- FAIL: TestLogGrowthWhenRefreshingPendingCommands/proposeOnFollower=true (1774.89s)

Parameters:

• attempt=1
• race=true
• run=3
• shard=4

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-45919

[miraradeva]

I can't tell if this is failing for the same reason as the CI for the test fix in #137441. The failure is different from the one previously on master #136426. The fix was also not backported to 24.3

@iskettaneh and @arulajmani, I'm assigning to you to figure out if the backport will address this.

[iskettaneh]

I believe this is a failure not related to #137441 and not related to leader leases (it doesn't run leader leases in the test in 24.3).

[iskettaneh]

This happened ~6 months ago in #102121, and it was closed as expected since the circuit breaker got tripped