roachtest: kv/splits/nodes=3/quiesce=false/lease=leader failed

[cockroach-teamcity]

roachtest.kv/splits/nodes=3/quiesce=false/lease=leader failed with artifacts on master @ 47699f3887ad5d1b8c7c5905eb5c49628aa59bbe:

(cluster.go:2481).Run: full command output in run_075326.211381454_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=false/lease=leader/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-45903

[cockroach-teamcity]

roachtest.kv/splits/nodes=3/quiesce=false/lease=leader failed with artifacts on master @ 47699f3887ad5d1b8c7c5905eb5c49628aa59bbe:

(cluster.go:2481).Run: full command output in run_081944.177141830_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=false/lease=leader/cpu_arch=arm64/run_1

Parameters:

• arch=arm64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[miraradeva]

Nodes definitely became overloaded in both failures above.

In the first failure, overload started around 08:03 when the CPU hit 96-98% utilization. The number of ranges reached 79k (80k is the stopping point for the test).

In the second failure, the CPU peaked at 93-95%, but the number of split ranges reached only 62k.

We could lower the expected number of splits for this test, but I think I'd rather watch at what point it fails and observe the improvements as part of the LeadSupportUntil optimization (that Ibrahim is working on) and the store-liveness-informed Raft quiescence of followers (#133885).