Skip to content

e2e tests example #36

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

e2e tests example #36

wants to merge 6 commits into from

Conversation

Ubun1
Copy link
Contributor

@Ubun1 Ubun1 commented Apr 8, 2022

requested in #30

this is draft PR with e2e tests example based on VMware Fusion + vagrant + ansible + molecule.
One of best advantage of this solution that this is local first e2e tests.
But downside of this local first solution that this is incompatible with current CI (circleci).

With this PR it's possible to open discussion about e2e test for this project. I'm open to all comments)

Current env consist of:

  • ubuntu2010 vm (base)
  • installed libvirtd, podman inside vm
  • created vm (cirros) inside base vm (ubuntu2010)
  • cloned project sources inside base vm
  • built libvirt_exporter images and run libvirt_exporter container

e2e tests located in ansible/molecule/default/verify.yml

@AlexZzz
Copy link
Contributor

AlexZzz commented Apr 24, 2022

Thanks for the PR!

Unfortunately, I didn't have time to check how it works right now. But it seems like a good starting point.

I see two huge problems with e2e tests of this exporter. They come from it's essense:

  1. This exporter requires libvirt daemon and virtual machine (or machines?) running. It also requires access to libvirtd socket available. I don't know any public CI solution with nested virtualization available.
  2. In the best-case scenario tests should validate metrics. As a reference, we can take values from virsh command output. If virsh statistics shows the same results as libvirt-exporter (don't forget possible units change here! Ex: nanoseconds -> seconds, etc.) then tests seem passed. But how is that possible to avoid values drift which might (and will) happen between virsh command run and libvirt-exporter's metrics collection?

For the first problem: we may run a virtual machine in some cloud via API. Connect to that machine, start nested vm and run tests. I'm not sure how adequate this is. Seems like a lot of hard work, there's no need in such automation right now.

Also, I've heard that circle-ci will ban me if I log into it from Russian IP:) don't want to check it, but it's possible to mistakenly do it and ruin all the CI and tests.

All these complexity makes me think we don't need a CI-based tests for this tiny project. Local-only tests are more than OK. But, it's still interesting how is it possible to validate metrics values, maybe there's some easy-to-implement solution for this?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants