roachtest: decommission/mixed-versions failed #139413

cockroach-teamcity · 2025-01-19T07:38:25Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ 758fe6af0492e04f677196745f28a546b3a657cc:

(test_runner.go:1475).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
(test_runner.go:1475).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=separate-process
mvtVersions=v23.1.21 → v23.2.6 → v24.1.8 → v24.3.2 → master
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [C-test-failure O-roachtest O-robot T-kv branch-release-25.1 release-blocker]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv branch-release-24.3 release-blocker]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

Jira issue: CRDB-46640

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2025-01-20T07:28:33Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ 758fe6af0492e04f677196745f28a546b3a657cc:

(test_runner.go:1475).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
(test_runner.go:1475).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=separate-process
mvtVersions=v24.2.5 → v24.3.1 → master
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [C-test-failure O-roachtest O-robot T-kv branch-release-25.1 release-blocker]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv branch-release-24.3 release-blocker]

_{This test on roachdash | Improve this report!}

stevendanna · 2025-01-20T17:03:26Z

This looks like it could be an issue in the /health endpoint for tenants. Namely, the health endpoint is telling us that n1 is OK but it has actually been decomissioned.

stevendanna · 2025-01-20T17:05:31Z

My guess is that this isn't a new bug. Rather I think it might have been uncovered by a fix that allowed these post-assertion failures to run against secure and insecure clusters: 6efeadc (Edit: maybe not as we seem to also be seeing this on release branches)

cockroach-teamcity · 2025-01-25T07:13:08Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ 12a979cd5f2ac1d37502a1353596a1d77751ab5d:

(test_runner.go:1475).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
(test_runner.go:1475).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=separate-process
mvtVersions=v24.1.10 → v24.2.2 → v24.3.0 → master
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [A-kv-server C-bug C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-25.1]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.3 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2025-01-29T06:33:05Z

roachtest.decommission/mixed-versions failed with artifacts on master @ 11265b25b659c9858345b484f73740359a50613b:

(mixedversion.go:804).Run: mixed-version test failure while running step 4 (run "preload data"): failed to import fixtures: full command output in run_063233.975598097_n2_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommission/mixed-versions/cpu_arch=arm64/run_1

Parameters:

arch=arm64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=system-only
mvtVersions=v23.2.3 → v24.1.0 → v24.2.8 → v24.3.3 → master
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [A-kv-server C-bug C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-25.1]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.3 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2025-01-30T06:39:12Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ 8655f5db5fe8ff856d90f71211a3158f60f1769f:

(mixedversion.go:804).Run: mixed-version test failure while running step 9 (run "preload data"): failed to import fixtures: full command output in run_063838.641880966_n2_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=separate-process
mvtVersions=v24.1.3 → v24.2.4 → v24.3.2 → master
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [A-kv-server C-bug C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-25.1]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.3]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2025-02-04T07:03:50Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ 82b1a2b174dcedfd23b0a3a40c6069143f4b038f:

(test_runner.go:1475).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
(test_runner.go:1475).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=separate-process
mvtVersions=v23.2.5 → v24.1.10 → v24.2.3 → v24.3.2 → master
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [A-kv-server C-bug C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-25.1]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.3]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2025-02-08T06:59:50Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ bbaa2e50b2fe789527aac09b99fa5eee432e7695:

(test_runner.go:1478).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
(test_runner.go:1478).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=separate-process
mvtVersions=v23.2.12 → v24.1.6 → v24.3.0 → master
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #140774 roachtest: decommission/mixed-versions failed [C-test-failure O-roachtest O-robot T-kv branch-release-25.1.0-rc release-blocker]
roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [A-kv-server C-bug C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-25.1]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.3]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2025-02-09T06:57:15Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ ecfb74b7ad75327f11814966a2cdab1b9793a549:

(test_runner.go:1478).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
(test_runner.go:1478).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=separate-process
mvtVersions=v24.1.1 → v24.3.3 → master
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #140774 roachtest: decommission/mixed-versions failed [C-test-failure O-roachtest O-robot T-kv branch-release-25.1.0-rc release-blocker]
roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [A-kv-server C-bug C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-25.1]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.3]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2025-02-10T06:56:01Z

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ 5e499490a29405b71d5b2d568a43fb7e6b14fe56:

(test_runner.go:1478).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
(test_runner.go:1478).func1: failed during post test assertions (see test-post-assertions.log): pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=4
encrypted=false
mvtDeploymentMode=separate-process
mvtVersions=v24.1.7 → v24.3.4 → master
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: decommission/mixed-versions failed #140774 roachtest: decommission/mixed-versions failed [C-test-failure O-roachtest O-robot T-kv branch-release-25.1.0-rc release-blocker]
roachtest: decommission/mixed-versions failed #139411 roachtest: decommission/mixed-versions failed [A-kv-server C-bug C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-25.1]
roachtest: decommission/mixed-versions failed #139393 roachtest: decommission/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.3]

_{This test on roachdash | Improve this report!}

With this patch, at the end of decommissioning, we call the drain step as we would for `./cockroach node drain`: ``` [...] ..... id is_live replicas is_decommissioning membership is_draining readiness blocking_ranges 1 true 2 true decommissioning false ready 0 ..... id is_live replicas is_decommissioning membership is_draining readiness blocking_ranges 1 true 1 true decommissioning false ready 0 ...... id is_live replicas is_decommissioning membership is_draining readiness blocking_ranges 1 true 0 true decommissioning false ready 0 draining node n2 node is draining... remaining: 26 node is draining... remaining: 0 (complete) node n2 drained successfully No more data reported on target nodes. Please verify cluster health before removing the nodes. ``` In particular, note how the first invocation returns a RemainingIndicator of 26. This explains the failure in cockroachdb#140774 - cockroachdb#138732 was insufficient as it did not guarantee that the node had actually drained fully by the time it was marked as fully decommissioned and the `node decommission` had returned. See cockroachdb#140774. I verified that the modified decommission/drains roachtest passes via ``` ./pkg/cmd/roachtest/roachstress.sh -l -c 1 decommission/drains/alive ``` Touches cockroachdb#140774. ^-- backport to 25.1-rc would fix it. Touches cockroachdb#139411. ^-- backport to 25.1 will fix it. Fixes cockroachdb#139413. Release note (ops change): the node decommission cli command now waits until the target node is drained before marking it as fully decommissioned. Previously, it would start drain but not wait, leaving the target node briefly in a state where it would be unable to communicate with the cluster but would still accept client requests (which would then hang or hit unexpected errors). Note that a previous release note claimed to fix the same defect, but in fact only reduced the likelihood of its occurrence. As of this release note, this problem has truly been addressed. Epic: None

blathers-crl · 2025-02-13T11:35:06Z

cc @cockroachdb/test-eng

tbg · 2025-02-13T11:35:10Z

See #140774 (comment)

With this patch, at the end of decommissioning, we call the drain step as we would for `./cockroach node drain`: ``` [...] ..... id is_live replicas is_decommissioning membership is_draining readiness blocking_ranges 1 true 2 true decommissioning false ready 0 ..... id is_live replicas is_decommissioning membership is_draining readiness blocking_ranges 1 true 1 true decommissioning false ready 0 ...... id is_live replicas is_decommissioning membership is_draining readiness blocking_ranges 1 true 0 true decommissioning false ready 0 draining node n2 node is draining... remaining: 26 node is draining... remaining: 0 (complete) node n2 drained successfully No more data reported on target nodes. Please verify cluster health before removing the nodes. ``` In particular, note how the first invocation returns a RemainingIndicator of 26, so before this patch, we had initiated draining, but it hadn't fully completed. I thought for a while that this could explain cockroachdb#140774, i.e. that cockroachdb#138732 was insufficient as it did not guarantee that the node had actually drained fully by the time it was marked as fully decommissioned and the `node decommission` had returned. But I found that fully draining did not fix the test, and ultimately tracked the issue down to a test infra problem. Still, this PR is a good change, that brings the drain experience in decommission on par with the standalone CLI. See cockroachdb#140774. I verified that the modified decommission/drains roachtest passes via ``` ./pkg/cmd/roachtest/roachstress.sh -l -c 1 decommission/drains/alive ``` Touches cockroachdb#140774. Touches cockroachdb#139411. Touches cockroachdb#139413. PR cockroachdb#138732 already fixed most of the drain issues, but since the decommissioning process still went ahead and shut the node out from the cluster, SQL connections that drain was still waiting for would likely hit errors (since the gateway node would not be able to connect to the rest of the cluster any more due to having been flipped to fully decommissioned). So there's a new release note for the improvement in this PR, which avoids that. Release note (bug fix): previously, a node that was drained as part of decommissioning may have interrupted SQL connections that were still active during drain (and for which drain would have been expected to wait). Epic: None

stevendanna added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Jan 20, 2025

stevendanna added P-2 Issues/test failures with a fix SLA of 3 months and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 20, 2025

This was referenced Jan 20, 2025

roachtest: decommission/mixed-versions failed #139393

Closed

roachtest: decommission/mixed-versions failed #139411

Open

cockroach-teamcity mentioned this issue Feb 8, 2025

roachtest: decommission/mixed-versions failed #140774

Open

tbg self-assigned this Feb 13, 2025

tbg mentioned this issue Feb 13, 2025

cli: fully and properly drain target node of decommission #141411

Draft

tbg removed their assignment Feb 13, 2025

tbg added T-testeng TestEng Team and removed T-kv KV Team labels Feb 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: decommission/mixed-versions failed #139413

roachtest: decommission/mixed-versions failed #139413

cockroach-teamcity commented Jan 19, 2025 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Jan 20, 2025

stevendanna commented Jan 20, 2025

stevendanna commented Jan 20, 2025 •

edited

Loading

cockroach-teamcity commented Jan 25, 2025

cockroach-teamcity commented Jan 29, 2025

cockroach-teamcity commented Jan 30, 2025

cockroach-teamcity commented Feb 4, 2025

cockroach-teamcity commented Feb 8, 2025

cockroach-teamcity commented Feb 9, 2025

cockroach-teamcity commented Feb 10, 2025

blathers-crl bot commented Feb 13, 2025

tbg commented Feb 13, 2025

roachtest: decommission/mixed-versions failed #139413

roachtest: decommission/mixed-versions failed #139413

Comments

cockroach-teamcity commented Jan 19, 2025 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Jan 20, 2025

stevendanna commented Jan 20, 2025

stevendanna commented Jan 20, 2025 • edited Loading

cockroach-teamcity commented Jan 25, 2025

cockroach-teamcity commented Jan 29, 2025

cockroach-teamcity commented Jan 30, 2025

cockroach-teamcity commented Feb 4, 2025

cockroach-teamcity commented Feb 8, 2025

cockroach-teamcity commented Feb 9, 2025

cockroach-teamcity commented Feb 10, 2025

blathers-crl bot commented Feb 13, 2025

tbg commented Feb 13, 2025

cockroach-teamcity commented Jan 19, 2025 •

edited by cockroach-jira-scripts

Loading

stevendanna commented Jan 20, 2025 •

edited

Loading