DAOS-18387 test: recovery/ddb.py test_recovery_ddb_ls MD-on-SSD Support by shimizukko · Pull Request #17332 · daos-stack/daos

shimizukko · 2025-12-31T05:44:46Z

To support MD-on-SSD for ddb, we need to support two commands. ddb prov_mem and ddb ls with --db_path.

Update ddb_utils.py to support the new commands.

Update test_recovery_ddb_ls to support MD-on-SSD with the new ddb commands.

We need to update the test yaml to run on MD-on-SSD/HW Medium, but that will break other tests in ddb.py because they don't support MD-on-SSD yet. Keep the original tests as ddb_pmem.py and ddb_pmem.yaml and keep running them on VM (except test_recovery_ddb_ls because that's updated in this PR).

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium: false
Test-tag: test_recovery_ddb_ls DdbPMEMTest

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

To support MD-on-SSD for ddb, we need to support two commands. ddb prov_mem and ddb ls with --db_path. Update ddb_utils.py to support the new commands. Add check_ram_used in recovery_utils.py to detect whether the system is MD-on-SSD. Update test_recovery_ddb_ls to support MD-on-SSD with the new ddb commands. We need to update the test yaml to run on MD-on-SSD/HW Medium, but that will break other tests in ddb.py because they don't support MD-on-SSD yet. Keep the original tests as ddb_pmem.py and ddb_pmem.yaml and keep running them on VM (except test_recovery_ddb_ls because that's updated in this PR). Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

github-actions · 2025-12-31T05:45:04Z

Ticket title is 'CR Test Update - recovery/ddb.py test_recovery_ddb_ls MD-on-SSD Support'
Status is 'In Review'
Labels: 'catastrophic_recovery'
https://daosio.atlassian.net/browse/DAOS-18387

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

daosbuild3 · 2025-12-31T06:08:36Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17332/1/display/redirect

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

daosbuild3 · 2025-12-31T09:37:45Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17332/5/execution/node/857/log

daosbuild3 · 2025-12-31T10:08:06Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17332/5/execution/node/898/log

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

shimizukko · 2026-01-02T06:56:47Z

@phender @dinghwah
In Git diff, it looks like ddb_pmem.py and ddb_pmem.yaml are new, but they appear so because I removed test_recovery_ddb_ls from there and renamed the files by adding _pmem. Please focus on the rest of the files.

I have two questions:

In recovery_utils.py, I added check_ram_used to determine whether the system is MD-on-SSD. The logic is that if the test runs on HW Medium and the server config has ram field, it must be running on MD-on-SSD. Is this okay or is there any other better way?
On MD-on-SSD, we need to load/mount the pool dir to a new location. That location can be anywhere, but I chose /mnt/daos_load to make it consistent with the existing pattern. At the end of the test, I call umount and rm -rf on /mnt/daos_load. Is this okay?

Thanks.

src/tests/ftest/recovery/ddb.py

src/tests/ftest/util/recovery_utils.py

src/tests/ftest/recovery/ddb.py

src/tests/ftest/util/ddb_utils.py

src/tests/ftest/recovery/ddb.py

phender · 2026-01-05T16:00:47Z

src/tests/ftest/recovery/ddb_pmem.py

+    Args:
+        remote_file_path (str): File path to copy to local.
+        test_dir (str): Test directory. Usually self.test_dir.
+        remote (str): Remote hostname to copy file from.


get_clush_command requires a NodeSet.

Suggested change

remote (str): Remote hostname to copy file from.

remote (NodeSet): Remote hostname to copy file from.

Did the change not get committed?

I forgot to add this file. Fixed.

src/tests/ftest/recovery/ddb_pmem.py

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

Also use self.server_managers[0].manager.job.yaml.metadata_params.path.value to get control_metadata_path. Update self.fail log message to include failed hosts. Update comment. Include up-to-date sample output. Update test yaml timeout value. Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

phender · 2026-02-04T14:26:35Z

src/tests/ftest/recovery/ddb.py

+            vos_paths = self.server_managers[0].get_vos_files(pool)
+            if not vos_paths:
+                self.fail(
+                    f"vos file wasn't found in {self.server_managers[0].get_vos_paths(pool)[0]}")


Why would we call get_vos_files() a second time for this error message, if it returned an empty list the first time we called it?

I don't remember why I wrote this. Fixed.

phender · 2026-02-04T14:29:59Z

src/tests/ftest/recovery/ddb.yaml

  test_clients: 1

-timeout: 1800
+timeout: 30M


Based upon https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17332/12/testReport/FTEST_recovery/DdbTest/ should this be 6 minutes?

Suggested change

timeout: 30M

timeout: 360

In my FTC node, it takes much longer than 6 min, so I put 30 min. Maybe it's slow if the node doesn't have PMEM? I adjusted it to the reasonable value for CI.

phender · 2026-02-04T14:35:17Z

src/tests/ftest/recovery/ddb_pmem.yaml

+  test_servers: 1
+  test_clients: 1
+
+timeout: 1800


Based upon https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17332/12/testReport/FTEST_recovery/DdbPMEMTest/ should this be 6 minutes?

Suggested change

timeout: 1800

timeout: 360

phender · 2026-02-04T14:40:26Z

src/tests/ftest/recovery/ddb_pmem.py

+    Args:
+        remote_file_path (str): File path to copy to local.
+        test_dir (str): Test directory. Usually self.test_dir.
+        remote (str): Remote hostname to copy file from.


Did the change not get committed?

src/tests/ftest/recovery/ddb.py

daltonbohning · 2026-02-04T18:30:13Z

src/tests/ftest/recovery/ddb_pmem.py

@@ -0,0 +1,468 @@
+"""
+  (C) Copyright 2022-2024 Intel Corporation.


Seems this copyright is wrong / copy-pasted?

I thought we leave the Intel Copyright statement and add HPE below it.

But this is a new file, right? So there was no work done on it at Intel

Oh, I see in your other comment you moved it

ddb_pmem.py used to be ddb.py

That wasn't obvious to me to GitHub's diff

Yes, so I mentioned that in the comment above, but didn't include you in the at mention.

daltonbohning · 2026-02-04T18:32:19Z

src/tests/ftest/recovery/ddb_pmem.py

+        self.random_akey = get_random_string(10)
+        self.random_data = get_random_string(10)
+
+    def test_recovery_ddb_rm(self):


In general we should use log_step in these new tests

ddb_pmem.py used to be ddb.py. I added MD-on-SSD support for test_recovery_ddb_ls, but then I had to also update the test yaml to support MD-on-SSD, so I renamed the original test to ddb_pmem.py and moved test_recovery_ddb_ls to ddb.py, which now supports both modes. I'll move the remaining tests in ddb_pmem.py to ddb.py and add MD-on-SSD support. At that time, I want to do a bunch of refactoring including log_step.

You're just moving these temporarily to ddb_pmem.py and then plan to move them back? Is it too much work to make them all work now?

I refactored ddb_pmem.py to include log_step, etc.

Remove get_vos_paths() from self.fail. Reduce timeout to 7M. Use single quotes to surround double quotes. Move DdbCommand instantiation outside of if-else. Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

daltonbohning · 2026-02-09T21:09:40Z

src/tests/ftest/util/ddb_utils.py

+        Before calling this method, "" (two double quotes) needs to be set to
+        self.vos_path.


I don't understand this comment. Why can't this function handle that?

This is my opinion.

If we do self.vos_path.value = '""' here, vos_path = '""' in ddb.py (line 138) would be unnecessary, so we should remove that. In that case, what do we set to vos_path when we instantiate DdbCommand? Setting None and letting prov_mem() update it to '""' later seems odd. I meant the comment as instruction rather than requirement.

Why does it need to be set to '""' to begin with? Does the command itself require an empty path? That seems odd

Yes, the command itself requires an empty path. I agree it's odd.

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

shimizukko · 2026-02-16T07:58:31Z

@daos-stack/daos-gatekeeper I updated the commit message, but if we simply merge it, I believe the old message will be used. Please use the one shown above when merging.

DAOS-18387 test: Fix pylint

975427f

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

shimizukko added 3 commits December 31, 2025 06:15

DAOS-18387 test: Fix pylint

70db493

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

DAOS-18387 test: Fix pylint

517afe8

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

DAOS-18387 test: Fix pylint

5327dc4

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

shimizukko added 3 commits January 2, 2026 05:13

Merge branch 'master' into makito/DAOS-18387

f40059b

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

DAOS-18387 test: Add targets: 1 to ddb.yaml

069c126

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

DAOS-18387 test: Fix pylint

a4106c0

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

shimizukko marked this pull request as ready for review January 2, 2026 06:44

shimizukko requested review from a team as code owners January 2, 2026 06:44

shimizukko requested review from dinghwah and phender January 2, 2026 06:44

phender requested changes Jan 5, 2026

View reviewed changes

shimizukko added 5 commits February 3, 2026 06:17

Merge branch 'master' into makito/DAOS-18387

dd90321

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

Merge branch 'master' into makito/DAOS-18387

ac94bb2

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

DAOS-18387 test: Fix typo

9806f88

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

DAOS-18387 test: Fix pylint

ab97d3d

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

shimizukko requested a review from phender February 4, 2026 05:47

phender requested changes Feb 4, 2026

View reviewed changes

daltonbohning reviewed Feb 4, 2026

View reviewed changes

shimizukko added 3 commits February 5, 2026 05:46

Merge branch 'master' into makito/DAOS-18387

206d0e4

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

DAOS-18387 test: Fix pylint

70ac162

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

shimizukko requested review from daltonbohning and phender February 5, 2026 11:02

shimizukko added 3 commits February 6, 2026 00:47

Merge branch 'master' into makito/DAOS-18387

d5f6755

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

DAOS-18387 test: Refactor ddb_pmem.py

f1d996f

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

DAOS-18387 test: Fix pylint

042df90

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

daltonbohning reviewed Feb 9, 2026

View reviewed changes

shimizukko added 2 commits February 10, 2026 23:19

Merge branch 'master' into makito/DAOS-18387

c05349c

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest

DAOS-18387 test: Set "" to vos_path in prov_mem()

a81f1c1

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium: false Test-tag: test_recovery_ddb_ls DdbPMEMTest Signed-off-by: Makito Kano <makito.kano@hpe.com>

shimizukko requested a review from daltonbohning February 11, 2026 01:13

daltonbohning approved these changes Feb 11, 2026

View reviewed changes

dinghwah approved these changes Feb 16, 2026

View reviewed changes

shimizukko requested a review from a team February 16, 2026 07:56

	remote (str): Remote hostname to copy file from.
	remote (NodeSet): Remote hostname to copy file from.

		@@ -0,0 +1,468 @@
		"""
		(C) Copyright 2022-2024 Intel Corporation.

		Before calling this method, "" (two double quotes) needs to be set to
		self.vos_path.

Conversation

shimizukko commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

shimizukko commented Jan 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shimizukko commented Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

shimizukko commented Dec 31, 2025 •

edited

Loading

github-actions bot commented Dec 31, 2025 •

edited

Loading