From d4a06bdae90a64e85aee7e8206e23fd1b27794db Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Tue, 7 Mar 2023 19:12:35 +0100 Subject: [PATCH 01/10] Testing responsibilies initial creation Signed-off-by: Marek Aufart --- .../testing/testing-responsibilities.md | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 enhancements/tackle/testing/testing-responsibilities.md diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md new file mode 100644 index 00000000..6e10b878 --- /dev/null +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -0,0 +1,92 @@ +--- +title: testing-responsibilities +authors: + - "@aufi" +reviewers: + - TBD +approvers: + - TBD +creation-date: "2023-03-07" +last-updated: "2023-03-07" +status: provisional|implementable|implemented|deferred|rejected|withdrawn|replaced +see-also: + - "/enhancements/testing/high-level-summary.md" + - https://github.com/konveyor/enhancements/pull/98 +--- + +# Konveyor testing responsibilities + +## Release Signoff Checklist + +- [ ] Enhancement is `implementable` +- [ ] Design details are appropriately documented from clear requirements +- [ ] Test plan is defined +- [ ] User-facing documentation is created + +## Summary + +Testing of Konveyor is critical to keep its quality, release stability and happyness of community. In order to accomplish that, it is needed to create, execute and maintain several kinds of tests like end-to-end and components-specific test suites. + +This enhancement should provide an introduction to Konveyor project testing strategy and define basic responsibilities for community members including developers as well as QE. + +## Motivation + +As we build an upstream community project that consists of multiple components, it is needed to ensure quality and stability of the project. One of important parts to achieve this is to have a test suite. + +There are multiple level of tests like unit, integration or end-to-end, those could be executed on several levels from a single component unit tests to the Konveyor end-to-end UI test suite executed by automated CI. + +The motivation for this enhancement is to do it in the most effective and engineers-friendly way. + +### Goals + +The main goal is to have a consistent test suite covering most of Konveyor application features that can be executed automatically and together with component tests, results will be reported to Konveyor CI. + +An important part of this effort is to set expectations and basic responsibilities for Konveyor upstream community members regarding to test creation, execution and maintenance. + +### Non-Goals + +Cover downstream testing. + +## Proposal + + + + + +### Implementation Details/Notes/Constraints [optional] + +technologies + +What are the caveats to the implementation? What are some important details that +didn't come across above. Go in to as much detail as necessary here. This might +be a good place to talk about core concepts and how they relate. + +### Security, Risks, and Mitigations + +Upstream test suite must not contain any company internal information not customer data (not even as sample data). + +## Design Details + +### Test Plan + +Open Konveyor upstream CI at https://github.com/konveyor/ci and see its status (green hopefully). + +### Upgrade / Downgrade Strategy + +Tests are dependent on Konveyor application, so might need to be branched/tagged and executed together and specifically for given Konveyor version. + +No upgrade/downgrade actions for upstream test suite are expected. + +## Implementation History + +This is a follow-up on Dylan's Testing overview enhancement https://github.com/konveyor/enhancements/pull/98. + +## Drawbacks and Alternatives + +Since testing could be considered as QE's responsibility, developers and Konveyor component maintainers don't need to care, so we might: +- not run upstream tests, leave it for downstream product builders OR +- not formalize upstream testing responsibilities too much to not make it over-engineered. + +## Infrastructure Needed + +Github actions with Minikube should be enough for upstream tests executions. From 760405e4945775b3beadcf0b6ff0c64e2363e630 Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Wed, 8 Mar 2023 15:30:43 +0100 Subject: [PATCH 02/10] Update responsibity matrix Signed-off-by: Marek Aufart --- .../testing/testing-responsibilities.md | 53 +++++++++++++++++-- 1 file changed, 48 insertions(+), 5 deletions(-) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index 6e10b878..41d2e0c3 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -49,17 +49,60 @@ Cover downstream testing. ## Proposal +Let's start with kinds of tests relevant for us: +- Component/repository specific **unit tests** + - Tests **logic** of a single Konveyor component (hub, UI, …), doesn’t require running whole Konveyor installation or other components (like Keycloak) to be executed, but either doesn’t need it or mocking it. + - Test technology/framework depends on technology used in a given component. +- Component/repository specific **integration tests** + - Tests **logic** of a single Konveyor component (hub, UI, …), but requires other Konveyor components to be running in order to pass the test (e.g. testing addon via requests to Hub to trigger analysis). +- Konveyor suite **end-to-end tests** (with UI or API) + - Tests basic **use-cases** of the Konveyor tool (scenarios which real users would expect to be working). + - Requires running Konveyor installation and covers functions of multiple components. +These tests should be executed and maintained as described in following section. +### Responsibilities +#### Overview matrix -### Implementation Details/Notes/Constraints [optional] +| Kind of test | Primary Responsible | Presence | Executed on | Executed from | Source code in | +|---|---|---|---|---|---| +| **unit** | Developers | optional | Component | PRs&push | Component repo | +| **integration** | Developers | required | Component+Konveyor | PRs&push | Component repo | +| **E2E** | QE | required | Running Konveyor | time-based schedule | E2E test suite repos | + +#### More specific matrix as a starting point for Konveyor Hub and E2E tests + +| Kind of test | Primary Responsible | Description | Source code | Triggered by | +|---|---|---|---|---| +| **integration** Hub | Hub developers | REST API coverage tests, maybe addons, [more](https://github.com/konveyor/tackle2-hub/discussions/241) | https://github.com/konveyor/tackle2-hub/... | PR&push | +| **integration** addon-windup | Addon developers | Bash-scripted windup analysis | https://github.com/konveyor/tackle2-addon-windup/blob/main/hack/test-e2e.sh | PR&push | +| **E2E** API | Developers&QE | Golang API test suite (WIP) | https://github.com/konveyor/go-konveyor-tests | time-based schedule | +| **E2E** UI | QE | Existing QE-maintained UI test suite using cypress framework | https://github.com/konveyor/tackle-ui-tests | time-based schedule | + + + +### What Konveyor org expects from its components +- Decide if unit tests are relevant for given component, if so, write it and maintain it. +- Be primarily responsible for maintaining component integration tests. +- Setup test actions on their components repositories and report it to Konveyor CI repo. -technologies +### What Konveyor components should expect from Konveyor org +- Provide tools for automated Konveyor installation setup (locally as well as with github actions) ready for running their integration tests on PRs (use Konveyor CI repo as a starting point). +- Provide working&stable Konveyor builds (partially ensured by e2e tests). + +### Execution and project status + +Konveyor CI repository is at https://github.com/konveyor/ci, it doesn't execute test suites itself, it just display status of E2E or components tests runs. + +Overall CI status is: +- GREEN if all E2E test suites and all reporting components tests are passing. +- YELLOW if all E2E test suites are passing, but some of components reporting tests are failing. +- RED if some of E2E test suites is failing. + +### Implementation Details/Notes/Constraints [optional] -What are the caveats to the implementation? What are some important details that -didn't come across above. Go in to as much detail as necessary here. This might -be a good place to talk about core concepts and how they relate. +TBD technologies&resources to for re-use ### Security, Risks, and Mitigations From 1b04c3773fb1da1df0fef7fb0a3b96fda728da40 Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Wed, 8 Mar 2023 16:28:04 +0100 Subject: [PATCH 03/10] Add notes component vs. E2E Signed-off-by: Marek Aufart --- enhancements/tackle/testing/testing-responsibilities.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index 41d2e0c3..6c57adc5 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -59,6 +59,8 @@ Let's start with kinds of tests relevant for us: - Tests basic **use-cases** of the Konveyor tool (scenarios which real users would expect to be working). - Requires running Konveyor installation and covers functions of multiple components. +TODO: That make a difference on components tests vs. E2E tests - devs vs QE - code logic tests vs end user tests scenarios. + These tests should be executed and maintained as described in following section. ### Responsibilities @@ -80,7 +82,7 @@ These tests should be executed and maintained as described in following section. | **E2E** API | Developers&QE | Golang API test suite (WIP) | https://github.com/konveyor/go-konveyor-tests | time-based schedule | | **E2E** UI | QE | Existing QE-maintained UI test suite using cypress framework | https://github.com/konveyor/tackle-ui-tests | time-based schedule | - +TODO:define what to test - like require QE to have links to test cases, devs to tickets/issues/enhancements ### What Konveyor org expects from its components - Decide if unit tests are relevant for given component, if so, write it and maintain it. From 602eee34a32dc11c07b71967306690508305d51d Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Wed, 8 Mar 2023 16:30:17 +0100 Subject: [PATCH 04/10] Update table Signed-off-by: Marek Aufart --- enhancements/tackle/testing/testing-responsibilities.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index 6c57adc5..ff8c313d 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -79,7 +79,8 @@ These tests should be executed and maintained as described in following section. |---|---|---|---|---| | **integration** Hub | Hub developers | REST API coverage tests, maybe addons, [more](https://github.com/konveyor/tackle2-hub/discussions/241) | https://github.com/konveyor/tackle2-hub/... | PR&push | | **integration** addon-windup | Addon developers | Bash-scripted windup analysis | https://github.com/konveyor/tackle2-addon-windup/blob/main/hack/test-e2e.sh | PR&push | -| **E2E** API | Developers&QE | Golang API test suite (WIP) | https://github.com/konveyor/go-konveyor-tests | time-based schedule | +|||||| +| **E2E** API | QE&Developers | Golang API test suite (WIP) | https://github.com/konveyor/go-konveyor-tests | time-based schedule | | **E2E** UI | QE | Existing QE-maintained UI test suite using cypress framework | https://github.com/konveyor/tackle-ui-tests | time-based schedule | TODO:define what to test - like require QE to have links to test cases, devs to tickets/issues/enhancements From 48bfd36df8d29dd4c9f9b590e10e6270b2602c4f Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Thu, 9 Mar 2023 10:23:12 +0100 Subject: [PATCH 05/10] Elaborate on integr. e2e difference Signed-off-by: Marek Aufart --- .../testing/testing-responsibilities.md | 43 ++++++++++++------- 1 file changed, 28 insertions(+), 15 deletions(-) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index ff8c313d..d28faefc 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -54,20 +54,33 @@ Let's start with kinds of tests relevant for us: - Tests **logic** of a single Konveyor component (hub, UI, …), doesn’t require running whole Konveyor installation or other components (like Keycloak) to be executed, but either doesn’t need it or mocking it. - Test technology/framework depends on technology used in a given component. - Component/repository specific **integration tests** - - Tests **logic** of a single Konveyor component (hub, UI, …), but requires other Konveyor components to be running in order to pass the test (e.g. testing addon via requests to Hub to trigger analysis). + - Tests **logic** of features from a single Konveyor component (hub, UI, …), but requires other Konveyor components to be running in order to pass the test (e.g. testing addon via requests to Hub to trigger analysis). - Konveyor suite **end-to-end tests** (with UI or API) - - Tests basic **use-cases** of the Konveyor tool (scenarios which real users would expect to be working). + - Tests basic **use-cases** of the Konveyor tool (scenarios which real users would expect to be working, ideally matching to Polarion test cases). - Requires running Konveyor installation and covers functions of multiple components. -TODO: That make a difference on components tests vs. E2E tests - devs vs QE - code logic tests vs end user tests scenarios. - These tests should be executed and maintained as described in following section. +### Why such structure + +Component integration tests and E2E (API) tests are technically nearly the same, but they serves to slightly different purposes. At some point, PRs with new feature or fix should contain a test, the test written by the PR author should prove it basically works. There might be multiple PRs across Konveyor components for given feature, each component might test just it's part (focusing if it works, don't have to care too much about setting up other data to real use-case scenario). So, even Hub might have integration test using an addon, it might not care about all different options of using the feature (e.g. for an analysis, if the setup options like RWX, different kinds of identities, etc. might matter), but that's a stuff which developer's integration tests don't have to care much. + +Once the feature backend/API work was (mostly) completed, QE comes to play writing tests for it. They might use similar/shared methods with integration tests, but the tests focus on building real user test flows with relevant test data matching to Polarion test steps (if possible). + +An ideal workflow on developers&QE cooperation on a new feature work: + +|Feature started ->||||| +|---|---|---|---|---| +|Backend dev|Push PRs to e.g.addon/analyzer including tests
Push PRs to Hub including tests|||| +|UI dev||Make UI for the feature||| +|QE||Work on test steps&E2E API test|Write UI tests
Run sanity checks|| +|||||-> Feature ready for release process| + ### Responsibilities #### Overview matrix -| Kind of test | Primary Responsible | Presence | Executed on | Executed from | Source code in | +| Kind of test | Primary Responsible | Presence | Executed on | Trigger (min.required) | Source code in | |---|---|---|---|---|---| | **unit** | Developers | optional | Component | PRs&push | Component repo | | **integration** | Developers | required | Component+Konveyor | PRs&push | Component repo | @@ -75,16 +88,14 @@ These tests should be executed and maintained as described in following section. #### More specific matrix as a starting point for Konveyor Hub and E2E tests -| Kind of test | Primary Responsible | Description | Source code | Triggered by | +| Kind of test | Primary Responsible | Description | Tests code | Trigger (min.required) | |---|---|---|---|---| -| **integration** Hub | Hub developers | REST API coverage tests, maybe addons, [more](https://github.com/konveyor/tackle2-hub/discussions/241) | https://github.com/konveyor/tackle2-hub/... | PR&push | -| **integration** addon-windup | Addon developers | Bash-scripted windup analysis | https://github.com/konveyor/tackle2-addon-windup/blob/main/hack/test-e2e.sh | PR&push | +| **integration** Hub | Hub developers | REST API coverage tests, applications import, [more](https://github.com/konveyor/tackle2-hub/discussions/241) | https://github.com/konveyor/tackle2-hub/... | PR&push (+time anyway?) | +| **integration** addon-windup | Addon developers | Bash-scripted windup analysis | https://github.com/konveyor/tackle2-addon-windup/blob/main/hack/test-e2e.sh | PR&push (+time anyway?) | |||||| -| **E2E** API | QE&Developers | Golang API test suite (WIP) | https://github.com/konveyor/go-konveyor-tests | time-based schedule | +| **E2E** API | QE&Developers | Golang API test suite (WIP), focusing on sanity checks | https://github.com/konveyor/go-konveyor-tests | time-based schedule | | **E2E** UI | QE | Existing QE-maintained UI test suite using cypress framework | https://github.com/konveyor/tackle-ui-tests | time-based schedule | -TODO:define what to test - like require QE to have links to test cases, devs to tickets/issues/enhancements - ### What Konveyor org expects from its components - Decide if unit tests are relevant for given component, if so, write it and maintain it. - Be primarily responsible for maintaining component integration tests. @@ -96,7 +107,9 @@ TODO:define what to test - like require QE to have links to test cases, devs to ### Execution and project status -Konveyor CI repository is at https://github.com/konveyor/ci, it doesn't execute test suites itself, it just display status of E2E or components tests runs. +Konveyor CI repository is https://github.com/konveyor/ci + +The CI repo doesn't execute test suites itself, it just display status of E2E or components tests runs. Overall CI status is: - GREEN if all E2E test suites and all reporting components tests are passing. @@ -109,13 +122,13 @@ TBD technologies&resources to for re-use ### Security, Risks, and Mitigations -Upstream test suite must not contain any company internal information not customer data (not even as sample data). +Upstream test suite must not contain any company internal information not customer data (not even as a sample data). ## Design Details ### Test Plan -Open Konveyor upstream CI at https://github.com/konveyor/ci and see its status (green hopefully). +Open Konveyor upstream CI at https://github.com/konveyor/ci and tests statuses should be visible (green-ish hopefully). ### Upgrade / Downgrade Strategy @@ -131,7 +144,7 @@ This is a follow-up on Dylan's Testing overview enhancement https://github.com/k Since testing could be considered as QE's responsibility, developers and Konveyor component maintainers don't need to care, so we might: - not run upstream tests, leave it for downstream product builders OR -- not formalize upstream testing responsibilities too much to not make it over-engineered. +- not formalize upstream testing responsibilities too much to not make it over-engineered, just put some integration tests to Hub. ## Infrastructure Needed From 8d643bdb2296d4d96645548649db6aa6bbcf65ea Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Thu, 9 Mar 2023 15:21:04 +0100 Subject: [PATCH 06/10] Describe E2E as application level tests Signed-off-by: Marek Aufart --- enhancements/tackle/testing/testing-responsibilities.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index d28faefc..3548ca3f 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -55,15 +55,17 @@ Let's start with kinds of tests relevant for us: - Test technology/framework depends on technology used in a given component. - Component/repository specific **integration tests** - Tests **logic** of features from a single Konveyor component (hub, UI, …), but requires other Konveyor components to be running in order to pass the test (e.g. testing addon via requests to Hub to trigger analysis). -- Konveyor suite **end-to-end tests** (with UI or API) - - Tests basic **use-cases** of the Konveyor tool (scenarios which real users would expect to be working, ideally matching to Polarion test cases). +- Konveyor suite **end-to-end tests** (with UI or API on application level) + - Tests basic **use-cases** of the Konveyor application (scenarios which real users would expect to be working, ideally matching to Polarion test cases). - Requires running Konveyor installation and covers functions of multiple components. These tests should be executed and maintained as described in following section. ### Why such structure -Component integration tests and E2E (API) tests are technically nearly the same, but they serves to slightly different purposes. At some point, PRs with new feature or fix should contain a test, the test written by the PR author should prove it basically works. There might be multiple PRs across Konveyor components for given feature, each component might test just it's part (focusing if it works, don't have to care too much about setting up other data to real use-case scenario). So, even Hub might have integration test using an addon, it might not care about all different options of using the feature (e.g. for an analysis, if the setup options like RWX, different kinds of identities, etc. might matter), but that's a stuff which developer's integration tests don't have to care much. +Component integration tests and E2E (API) tests are technically nearly the same, but they serves to slightly different purposes. At some point, PRs with new feature or fix should contain a test, the test written by the PR author should prove it basically works. There might be multiple PRs across Konveyor components for given feature, each component might test just it's part (focusing if it works, don't have to care too much about setting up other data to real use-case scenario). + +So, even Hub might have integration test using an addon, it might not care about all different options of using the feature (e.g. for an analysis, if the setup options like RWX, different kinds of identities, etc. might matter), but that's a stuff which developer's integration tests don't have to care much. Once the feature backend/API work was (mostly) completed, QE comes to play writing tests for it. They might use similar/shared methods with integration tests, but the tests focus on building real user test flows with relevant test data matching to Polarion test steps (if possible). From 158a97f566f16a6fca118d840df6cf5ddada29fd Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Fri, 10 Mar 2023 16:27:37 +0100 Subject: [PATCH 07/10] Update E2E->application level E2E and user focus Signed-off-by: Marek Aufart --- .../testing/testing-responsibilities.md | 34 ++++++++++++------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index 3548ca3f..764d96c1 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -25,7 +25,7 @@ see-also: ## Summary -Testing of Konveyor is critical to keep its quality, release stability and happyness of community. In order to accomplish that, it is needed to create, execute and maintain several kinds of tests like end-to-end and components-specific test suites. +Testing of Konveyor is critical to keep its quality, release stability and happiness of community. In order to accomplish that, it is needed to create, execute and maintain several kinds of tests like application level end-to-end and components-specific test suites. This enhancement should provide an introduction to Konveyor project testing strategy and define basic responsibilities for community members including developers as well as QE. @@ -35,7 +35,7 @@ As we build an upstream community project that consists of multiple components, There are multiple level of tests like unit, integration or end-to-end, those could be executed on several levels from a single component unit tests to the Konveyor end-to-end UI test suite executed by automated CI. -The motivation for this enhancement is to do it in the most effective and engineers-friendly way. +The motivation for this enhancement is to do it in the most effective, user-focused and engineers-friendly way. ### Goals @@ -53,17 +53,24 @@ Let's start with kinds of tests relevant for us: - Component/repository specific **unit tests** - Tests **logic** of a single Konveyor component (hub, UI, …), doesn’t require running whole Konveyor installation or other components (like Keycloak) to be executed, but either doesn’t need it or mocking it. - Test technology/framework depends on technology used in a given component. + - Unit tests are optional and if existed should be placed in the component repository. + - Component/repository specific **integration tests** - Tests **logic** of features from a single Konveyor component (hub, UI, …), but requires other Konveyor components to be running in order to pass the test (e.g. testing addon via requests to Hub to trigger analysis). -- Konveyor suite **end-to-end tests** (with UI or API on application level) + - Tests should be placed in the component repository. Even the test might require other Konveyor components to test given component (e.g. addon might execute integration test sending request to Hub API which triggers the tested addon action, so it is practically an end-to-end test), the main point here is to test the component, other required Konveyor components are considered to be just dependencies required to run the test. + - As these tests focus on basic feature logic, assertions like expected HTTP response code, only partial check of returned data, etc. might be acceptable here. + +- Konveyor **application level end-to-end tests** (with UI or API) - Tests basic **use-cases** of the Konveyor application (scenarios which real users would expect to be working, ideally matching to Polarion test cases). - Requires running Konveyor installation and covers functions of multiple components. + - Test scenarios should follow expected usage of the Konveyor by its users. + - Assertions in tests require higher certainity that application works from user point of view than integration tests. E.g. Application creation test might look on HTTP response code too, but must try to retrieve the created application by its returned ID as a check if it was really created. These tests should be executed and maintained as described in following section. ### Why such structure -Component integration tests and E2E (API) tests are technically nearly the same, but they serves to slightly different purposes. At some point, PRs with new feature or fix should contain a test, the test written by the PR author should prove it basically works. There might be multiple PRs across Konveyor components for given feature, each component might test just it's part (focusing if it works, don't have to care too much about setting up other data to real use-case scenario). +Component integration tests and Application level E2E (API) tests are technically nearly the same, but they serves to slightly different purposes. At some point, PRs with new feature or fix should contain a test, the test written by the PR author should prove it basically works. There might be multiple PRs across Konveyor components for given feature, each component might test just it's part (focusing if it works, don't have to care too much about setting up other data to real use-case scenario). So, even Hub might have integration test using an addon, it might not care about all different options of using the feature (e.g. for an analysis, if the setup options like RWX, different kinds of identities, etc. might matter), but that's a stuff which developer's integration tests don't have to care much. @@ -75,7 +82,7 @@ An ideal workflow on developers&QE cooperation on a new feature work: |---|---|---|---|---| |Backend dev|Push PRs to e.g.addon/analyzer including tests
Push PRs to Hub including tests|||| |UI dev||Make UI for the feature||| -|QE||Work on test steps&E2E API test|Write UI tests
Run sanity checks|| +|QE||Work on test steps&application E2E API test|Write UI tests
Run sanity checks|| |||||-> Feature ready for release process| ### Responsibilities @@ -86,22 +93,23 @@ An ideal workflow on developers&QE cooperation on a new feature work: |---|---|---|---|---|---| | **unit** | Developers | optional | Component | PRs&push | Component repo | | **integration** | Developers | required | Component+Konveyor | PRs&push | Component repo | -| **E2E** | QE | required | Running Konveyor | time-based schedule | E2E test suite repos | +| **application E2E** | QE | required | Running Konveyor | time-based schedule | E2E test suite repos | #### More specific matrix as a starting point for Konveyor Hub and E2E tests -| Kind of test | Primary Responsible | Description | Tests code | Trigger (min.required) | +| Kind of test | Responsible | Description | Tests code | Trigger (min.required) | |---|---|---|---|---| | **integration** Hub | Hub developers | REST API coverage tests, applications import, [more](https://github.com/konveyor/tackle2-hub/discussions/241) | https://github.com/konveyor/tackle2-hub/... | PR&push (+time anyway?) | | **integration** addon-windup | Addon developers | Bash-scripted windup analysis | https://github.com/konveyor/tackle2-addon-windup/blob/main/hack/test-e2e.sh | PR&push (+time anyway?) | |||||| -| **E2E** API | QE&Developers | Golang API test suite (WIP), focusing on sanity checks | https://github.com/konveyor/go-konveyor-tests | time-based schedule | -| **E2E** UI | QE | Existing QE-maintained UI test suite using cypress framework | https://github.com/konveyor/tackle-ui-tests | time-based schedule | +| **application E2E** API | QE&Developers | Golang API test suite (WIP), focusing on sanity checks | https://github.com/konveyor/go-konveyor-tests | time-based schedule | +| **application E2E** UI | QE&Developers | Existing QE-maintained UI test suite using cypress framework | https://github.com/konveyor/tackle-ui-tests | time-based schedule | ### What Konveyor org expects from its components - Decide if unit tests are relevant for given component, if so, write it and maintain it. - Be primarily responsible for maintaining component integration tests. - Setup test actions on their components repositories and report it to Konveyor CI repo. +- For new implemented features, create E2E application level test (or work with QE on it) to ensure the feature is fully working and is ready for real-world usage. ### What Konveyor components should expect from Konveyor org - Provide tools for automated Konveyor installation setup (locally as well as with github actions) ready for running their integration tests on PRs (use Konveyor CI repo as a starting point). @@ -111,12 +119,12 @@ An ideal workflow on developers&QE cooperation on a new feature work: Konveyor CI repository is https://github.com/konveyor/ci -The CI repo doesn't execute test suites itself, it just display status of E2E or components tests runs. +The CI repo doesn't execute test suites itself, it just display status of application E2E or components tests runs. Overall CI status is: -- GREEN if all E2E test suites and all reporting components tests are passing. -- YELLOW if all E2E test suites are passing, but some of components reporting tests are failing. -- RED if some of E2E test suites is failing. +- GREEN if all application E2E test suites and all reporting components tests are passing. +- YELLOW if all application E2E test suites are passing, but some of components reporting tests are failing. +- RED if some of application E2E test suites is failing. ### Implementation Details/Notes/Constraints [optional] From 791d2421f4e1acc5e011e57cbf4af00461a1d6ae Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Mon, 13 Mar 2023 11:45:40 +0100 Subject: [PATCH 08/10] Add minimal alternative solution Signed-off-by: Marek Aufart --- enhancements/tackle/testing/testing-responsibilities.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index 764d96c1..14cb64c7 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -156,6 +156,12 @@ Since testing could be considered as QE's responsibility, developers and Konveyo - not run upstream tests, leave it for downstream product builders OR - not formalize upstream testing responsibilities too much to not make it over-engineered, just put some integration tests to Hub. +### A minimal working alternative + +As a minimal change to current state which would move us a step forward is: **Put integration (E2E) tests to Hub as a developers effort and start require tests on new features PRs.** + +Other things like Dev/QE cooperation on tests, application level API tests, requirements on type of tests, etc. would not be implemented. + ## Infrastructure Needed Github actions with Minikube should be enough for upstream tests executions. From 885ff4451a575d970b0426c85cdb87fe2c7375ef Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Tue, 14 Nov 2023 11:29:12 +0100 Subject: [PATCH 09/10] Update CI and outdated components Signed-off-by: Marek Aufart --- .../testing/testing-responsibilities.md | 29 +++++++------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index 14cb64c7..fb30db93 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -51,7 +51,7 @@ Cover downstream testing. Let's start with kinds of tests relevant for us: - Component/repository specific **unit tests** - - Tests **logic** of a single Konveyor component (hub, UI, …), doesn’t require running whole Konveyor installation or other components (like Keycloak) to be executed, but either doesn’t need it or mocking it. + - Tests **logic** of a single Konveyor component (hub, UI, …), doesn’t require running whole Konveyor installation or other components (like Keycloak SSO) to be executed, but either doesn’t need it or mocking it. - Test technology/framework depends on technology used in a given component. - Unit tests are optional and if existed should be placed in the component repository. @@ -91,19 +91,17 @@ An ideal workflow on developers&QE cooperation on a new feature work: | Kind of test | Primary Responsible | Presence | Executed on | Trigger (min.required) | Source code in | |---|---|---|---|---|---| -| **unit** | Developers | optional | Component | PRs&push | Component repo | -| **integration** | Developers | required | Component+Konveyor | PRs&push | Component repo | -| **application E2E** | QE | required | Running Konveyor | time-based schedule | E2E test suite repos | +| **unit** | Developers | optional | Component | PRs, push | Component repo | +| **integration** | Developers | required | Component+Konveyor | PRs, push | Component repo | +| **application E2E** | QE | required | Running Konveyor | PR, push, nightly | E2E test suite repos | #### More specific matrix as a starting point for Konveyor Hub and E2E tests | Kind of test | Responsible | Description | Tests code | Trigger (min.required) | |---|---|---|---|---| -| **integration** Hub | Hub developers | REST API coverage tests, applications import, [more](https://github.com/konveyor/tackle2-hub/discussions/241) | https://github.com/konveyor/tackle2-hub/... | PR&push (+time anyway?) | -| **integration** addon-windup | Addon developers | Bash-scripted windup analysis | https://github.com/konveyor/tackle2-addon-windup/blob/main/hack/test-e2e.sh | PR&push (+time anyway?) | -|||||| -| **application E2E** API | QE&Developers | Golang API test suite (WIP), focusing on sanity checks | https://github.com/konveyor/go-konveyor-tests | time-based schedule | -| **application E2E** UI | QE&Developers | Existing QE-maintained UI test suite using cypress framework | https://github.com/konveyor/tackle-ui-tests | time-based schedule | +| **integration** Hub | Hub developers | REST API coverage tests, applications import, [more](https://github.com/konveyor/tackle2-hub/discussions/241) | https://github.com/konveyor/tackle2-hub/... | PR, push, nightly | +| **application E2E** API | QE&Developers | Golang API test suite (WIP), focusing on sanity checks | https://github.com/konveyor/go-konveyor-tests | PR, push, nightly | +| **application E2E** UI | QE&Developers | Existing QE-maintained UI test suite using cypress framework | https://github.com/konveyor/tackle-ui-tests | PR, push, nightly | ### What Konveyor org expects from its components - Decide if unit tests are relevant for given component, if so, write it and maintain it. @@ -115,20 +113,15 @@ An ideal workflow on developers&QE cooperation on a new feature work: - Provide tools for automated Konveyor installation setup (locally as well as with github actions) ready for running their integration tests on PRs (use Konveyor CI repo as a starting point). - Provide working&stable Konveyor builds (partially ensured by e2e tests). -### Execution and project status +### Execution and project status - CI Konveyor CI repository is https://github.com/konveyor/ci -The CI repo doesn't execute test suites itself, it just display status of application E2E or components tests runs. +The CI repo doesn't execute test suites itself, but provides re-usable workflows that can orchestrate Konveyor build, execute relevant tests and display overall status. -Overall CI status is: -- GREEN if all application E2E test suites and all reporting components tests are passing. -- YELLOW if all application E2E test suites are passing, but some of components reporting tests are failing. -- RED if some of application E2E test suites is failing. +Part of the CI are also "gate" jobs for PRs in component repositories that executes tests on Konveyor built with given PR changes. -### Implementation Details/Notes/Constraints [optional] - -TBD technologies&resources to for re-use +The CI functionality should be provided for main as well as other supported release branches. ### Security, Risks, and Mitigations From df7e8834b612f005fdf6eaea540027df6f322e8d Mon Sep 17 00:00:00 2001 From: Marek Aufart Date: Wed, 15 Nov 2023 09:52:21 +0100 Subject: [PATCH 10/10] Mention test tiers Signed-off-by: Marek Aufart --- enhancements/tackle/testing/testing-responsibilities.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/enhancements/tackle/testing/testing-responsibilities.md b/enhancements/tackle/testing/testing-responsibilities.md index fb30db93..00c664aa 100644 --- a/enhancements/tackle/testing/testing-responsibilities.md +++ b/enhancements/tackle/testing/testing-responsibilities.md @@ -123,6 +123,12 @@ Part of the CI are also "gate" jobs for PRs in component repositories that execu The CI functionality should be provided for main as well as other supported release branches. +#### Test tiers + +Tests can be divided to and executed by _tiers_. The idea behind is to differentiate critical tests for Konveyor core functionality from test for unstable or experimental features. + +The most important tests are executed in _tier0_. Standard features test should be included in _tier1_ and _tier2_ contains test for unstable or experimental features. + ### Security, Risks, and Mitigations Upstream test suite must not contain any company internal information not customer data (not even as a sample data).