Improve testing infrastructure #1457
Description
Issue description
Why are some tests being skipped? Why are some tests expected to fail? Can we skip lit tests? If I mark something as skipped, how do I know if it is being skipped in the CI/CD? These are some questions that have been asked about the testing infrastructure. Some tasks may include:
- Conditionally executing lit tests for the standalone plugin based on the presence of the standalone plugin.
- Have separate directories for tests that are to be skipped under different conditions. We want all of them to be skipped or none of them to be skipped. This way executing tests with github actions should be all green.
- Error when expected failure no longer fails.