Make tests configurable #1884
Make tests configurable #1884
Conversation
|
Tests passed, but current code does no handle well invalid configuration. Will improve in the next push. |
|
Changes in last version:
|
We treated the deployment as generic deployment and passed many arguments hard coded in the test suite. This scheme will not work for adding new workloads. We need to move the workloads details to the implementation, and provide an easy way to create a workload with the generic options that can work for any workload. Add a NewDeployment function accepting a branch and a PVCSpec. The rest of the arguments (deployment name, app name, path) are constants in the deployment implementation. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
We want to make the tests configurable. To make it possible, we need to be able to create a workload from a string. The workloads package provides now: - New(name) - create a workload instance from name, or fail with helpful error message if the workload is unknown. - AvailableNames() - return list of workloads names. This is useful for providing useful error message or providing online help. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Similar to workloads, we need to create a deployer from a string. Add similar New() and AvailableNames() functions to make it possible. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The tests are defined as a list of yaml objects in the configuration file. We will generate the Go sub tests from this configuration. This changes resolve the issue of discovering the tests. The test are static configuration that is easy to inspect and modify as needed. Developer can create multiple configuration files to run subset of the tests, or additional tests that cannot run per-pull-request. We can use different configurations to do more extensive tests in daily or weekly runs, and quicker tests that run with every pull request. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
When reading the config, validate that: - Test workload, deployer, and pvcSpec are valid - There are no duplicate tests - There is at least one test With this change we can safely run all configured tests. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
|
Took initial look. It's okay. I would run once to understand more on e2e. But this change is fine for now. |
Generate and run the configured tests instead of hard coded test matrix. Fixes RamenDR#1615 Signed-off-by: Nir Soffer <nsoffer@redhat.com>
We have an easy way to discover the test since they are defined in the configuration file. Describe the configuration and rename the "List of tests" section to "Test summary", since the list of tests is now in the configuration file. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
|
Change in last version:
|
|
|
||
| var registry map[string]types.Deployer | ||
|
|
||
| func init() { |
There was a problem hiding this comment.
I think we should restrict the use of init() for truly init scenarios. Would have preferred a getOrBuildRegistry() function to maintain the registry which can be called wherever registry is required.
|
@nirs The changes are really good. One issue I notice is that running just the Validate test suite is not possible if the user doesn't specify the tests section in the config. Can you please create an issue for that?
|
Right, we can allow empty test list when parsing the config, and fail in the dr test if there are not tests. Added #1892 |
Make the tests easier to find and modify by moving them to the configuration file.
A new
testssections list all the tests:The e2e framework generates and run a Go sub test for every item in the test list.
The ramenctl test command will use the same configuration file to run subset of the tests on real clusters.
Fixes #1615