-
Notifications
You must be signed in to change notification settings - Fork 164
Open
Labels
admin: epicAn EPIC -- meta issue used to track a body of workAn EPIC -- meta issue used to track a body of workadmin: key-resultA key result (in the context of OKRs)A key result (in the context of OKRs)
Description
Problem
Currently, the e2e tests consists of the following steps (see tests/e2e/README.md for more details):
- start a docker container
- run a defined sequence of actions with expected states
- after each action, query the system state and compare against the expected state
Also, there are a set of predefined test cases (more than 20) that can be executed directly. Run go run ./tests/e2e/... --help
for details.
Although at the high level this e2e framework is good, there are several issues that need to be addressed:
- Some of the tests are non-deterministic as they depend on timing assumptions. As a result, tests are sometimes failing and need to be rerun.
- Most of the existing test cases have unnecessary actions resulting in tests with a high number of steps (e.g., the
happy-path
has 39 steps).- This makes the testing time long. As a result, the non-deterministic aspect has an even higher impact, e.g., if step 38 in the
happy-path
test case is randomly failing, the entire test case needs to be retried. - This also makes the tests complex and hard to modify and debug, e.g., there is no reason to start consumer chains or create IBC channels to test the consumer lifecycle on the provider.
- This makes the testing time long. As a result, the non-deterministic aspect has an even higher impact, e.g., if step 38 in the
- Many of the test cases are not part of the CI or the nightly tests, which means that are rarely run.
Closing criteria
The tasks below are completed.
Task list
### Must have
- [x] Create simpler actions, i.e., one task performed per action (e.g., start only the provider).
- [ ] Create EPIC with test cases that require both the provider and a consumer (e.g., CCV channel handshake)
- [ ] https://github.com/cosmos/interchain-security/issues/2315
- [ ] Create EPIC with test cases that require only the consumer (there might be none)
- [x] Replace long test cases with shorter ones that address a single concern
- [ ] Enable multiple test cases to run in parallel (one failing test case should not affect others)
- [ ] Remove test cases that are checking protocol logic (the scope of e2e testing is making sure the API works)
### Questions
- [ ] Is the interchain testing framework better?
- [ ] How can we reduce non-determinism in certain actions?
- [ ] What are the pain points with compatibility tests?
### Nice to have
- [ ] https://github.com/cosmos/interchain-security/issues/1325
- [ ] https://github.com/cosmos/interchain-security/issues/1487
### Others (to triage)
- [ ] https://github.com/cosmos/interchain-security/issues/2198
- [ ] https://github.com/cosmos/interchain-security/issues/1568
- [ ] https://github.com/cosmos/interchain-security/issues/1135
- [ ] https://github.com/cosmos/interchain-security/issues/1132
- [ ] https://github.com/cosmos/interchain-security/issues/877
- [ ] https://github.com/cosmos/interchain-security/issues/1296
Metadata
Metadata
Assignees
Labels
admin: epicAn EPIC -- meta issue used to track a body of workAn EPIC -- meta issue used to track a body of workadmin: key-resultA key result (in the context of OKRs)A key result (in the context of OKRs)