Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pytest runner mode #355

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Add pytest runner mode #355

wants to merge 14 commits into from

Conversation

mtreinish
Copy link
Owner

@mtreinish mtreinish commented Nov 5, 2023

This commit adds a new runner mode to stestr that enables users to run test suites using pytest instead of unittest. This is an opt-in feature as for compatibility reasons we'll always default to using unittest, as all existing stestr users have only been leveraging unittest to run tests. A pytest plugin that adds a subunit output mode is now bundled with stestr. When the user specifies running tests with pytest it calls out to pytest just as stestr does with the integrated unittest extension today and sets the appropriate flags to enable subunit output.

To facilitate this new feature pytest is added to the stestr requirements list. I debated making it an optional dependency, but to make it easier for the significantly larger pytest user base (it's downloaded ~900x more per month) it seemed simpler to just make it a hard requirement. But I'm still not 100% on this decision so if there is sufficient pushback we can start it out as an optional dependency.

Co-Authored-By: Joe Gordon [email protected]

Closes: #354

TODO:

  • Fix test failures
  • Add CI job to run existing tests using pytest mode (as pytest can run unittest)
  • Add dedicated integration tests written using pytest instead of unittest
  • Test various options work with pytest mode (I expect issue with some of the test selection stuff as test ids are different in pytest)

This commit adds a new runner mode to stestr that enables users to run
test suites using pytest instead of unittest. This is an opt-in feature
as for compatibility reasons we'll always default to using unittest, as
all existing stestr users have only been leveraging unittest to run
tests. A pytest plugin that adds a subunit output mode is now bundled
with stestr. When the user specifies running tests with pytest it calls
out to pytest just as stestr does with the integrated unittest extension
today and sets the appropriate flags to enable subunit output.

To facilitate this new feature pytest is added to the stestr
requirements list. I debated making it an optional dependency, but to
make it easier for the significantly larger pytest user base (it's
downloaded ~900x more per month) it seemed simpler to just make it a
hard requirement. But I'm still not 100% on this decision so if there
is sufficient pushback we can start it out as an optional dependency.

Co-Authored-By: Joe Gordon <[email protected]>

Closes: #354
@mtreinish mtreinish added the on hold When a PR is blocked and should not be merged label Nov 5, 2023
This commit adds a return code subclass that executes the tests in
pytest mode written with the pytest test suite. These tests caught a
couple of issues when running in pytest mode.
This commit adds a new CI job that runs stestr's unit test suite in
pytest mode instead of using stestr's built in subunit runner (based on
stdlib unittest).
This commit fixes the mocking used for the config file tests. It was
previously implicitly defining a runner argument as a mock object which
wasn't a valid choice for a config file to contain after parsing. This
commit just explicitly sets the mock to have the unittest runner set to
avoid failure.
@mtreinish mtreinish removed the on hold When a PR is blocked and should not be merged label Nov 11, 2023
@mtreinish mtreinish changed the title [WIP] Add pytest runner mode Add pytest runner mode Nov 11, 2023
@codecov-commenter
Copy link

codecov-commenter commented Nov 11, 2023

Codecov Report

Attention: Patch coverage is 10.45752% with 137 lines in your changes are missing coverage. Please review.

Project coverage is 59.18%. Comparing base (3862766) to head (8fd4105).

Current head 8fd4105 differs from pull request most recent head 705ad2f

Please upload reports for the commit 705ad2f to get more accurate results.

Files Patch % Lines
stestr/pytest_subunit.py 0.00% 124 Missing ⚠️
stestr/commands/run.py 18.18% 9 Missing ⚠️
stestr/config_file.py 77.77% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #355      +/-   ##
==========================================
- Coverage   61.95%   59.18%   -2.78%     
==========================================
  Files          30       31       +1     
  Lines        2618     2761     +143     
  Branches      426      463      +37     
==========================================
+ Hits         1622     1634      +12     
- Misses        877     1006     +129     
- Partials      119      121       +2     
Flag Coverage Δ
unittests 59.18% <10.45%> (-2.78%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mtreinish
Copy link
Owner Author

Hmm, testing on windows is failing: https://github.com/mtreinish/stestr/actions/runs/6835221394/job/18588774544?pr=355#step:6:18

!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

I'm not really sure what that error means. It is at least working fine on Linux and macOS.

The pytest subunit plugin was loading test paths incorrectly on windows
as a posix path. This was causing the test loading to fail unexpectedly
because pytest was unable to process the posix path version of a windows
path. This commit fixes this issue in the pytest plugin.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Pytest based test executor
2 participants