Skip to content

[Epic]: Refactor e2e tests to increase coverage and reduce complexity #2283

@mpoke

Description

@mpoke

Problem

Currently, the e2e tests consists of the following steps (see tests/e2e/README.md for more details):

  • start a docker container
  • run a defined sequence of actions with expected states
  • after each action, query the system state and compare against the expected state

Also, there are a set of predefined test cases (more than 20) that can be executed directly. Run go run ./tests/e2e/... --help for details.

Although at the high level this e2e framework is good, there are several issues that need to be addressed:

  • Some of the tests are non-deterministic as they depend on timing assumptions. As a result, tests are sometimes failing and need to be rerun.
  • Most of the existing test cases have unnecessary actions resulting in tests with a high number of steps (e.g., the happy-path has 39 steps).
    • This makes the testing time long. As a result, the non-deterministic aspect has an even higher impact, e.g., if step 38 in the happy-path test case is randomly failing, the entire test case needs to be retried.
    • This also makes the tests complex and hard to modify and debug, e.g., there is no reason to start consumer chains or create IBC channels to test the consumer lifecycle on the provider.
  • Many of the test cases are not part of the CI or the nightly tests, which means that are rarely run.

Closing criteria

The tasks below are completed.

Task list

### Must have
- [x] Create simpler actions, i.e., one task performed per action (e.g., start only the provider).
- [ ] Create EPIC with test cases that require both the provider and a consumer (e.g., CCV channel handshake)
- [ ] https://github.com/cosmos/interchain-security/issues/2315
- [ ] Create EPIC with test cases that require only the consumer (there might be none)
- [x] Replace long test cases with shorter ones that address a single concern
- [ ] Enable multiple test cases to run in parallel (one failing test case should not affect others)
- [ ] Remove test cases that are checking protocol logic (the scope of e2e testing is making sure the API works)
### Questions
- [ ] Is the interchain testing framework better?
- [ ] How can we reduce non-determinism in certain actions?
- [ ] What are the pain points with compatibility tests?
### Nice to have
- [ ] https://github.com/cosmos/interchain-security/issues/1325
- [ ] https://github.com/cosmos/interchain-security/issues/1487
### Others (to triage)
- [ ] https://github.com/cosmos/interchain-security/issues/2198
- [ ] https://github.com/cosmos/interchain-security/issues/1568
- [ ] https://github.com/cosmos/interchain-security/issues/1135
- [ ] https://github.com/cosmos/interchain-security/issues/1132
- [ ] https://github.com/cosmos/interchain-security/issues/877
- [ ] https://github.com/cosmos/interchain-security/issues/1296

Metadata

Metadata

Assignees

No one assigned

    Labels

    admin: epicAn EPIC -- meta issue used to track a body of workadmin: key-resultA key result (in the context of OKRs)

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions