Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect Flaky tests in an optimized way #456

Open
taman9333 opened this issue Dec 5, 2024 · 0 comments
Open

Detect Flaky tests in an optimized way #456

taman9333 opened this issue Dec 5, 2024 · 0 comments

Comments

@taman9333
Copy link

Right now, I am trying to detect flaky tests, but I think looping over the test_results.json file will be inefficient since I would need to go through all packages. However, I realized that the output of the gotestsum command lists the different packages being tested, along with whether each test succeeded or failed. At the end of the output, there is a === Failed section.

For example, the output in the case of the github-actions format might look like this:

::group::PASS cmd.TestStable (0.00s)
    main_test.go:9: This is a stable test

::endgroup::
::group::FAIL cmd.Test_Flaky2 (0.00s)
    main_test.go:21: Flaky test failed!

::endgroup::
  FAIL Package cmd (297ms)

  PASS mathutils.TestAdd (0.00s)
  PASS mathutils.TestMultiply (0.00s)
  PASS mathutils.Test_AddFlaky (0.00s)
  PASS Package mathutils (450ms)


DONE 5 tests, 1 failure in 1.011s

::group::FAIL cmd.Test_Flaky2 (re-run 1) (0.00s)
    main_test.go:21: Flaky test failed!

::endgroup::
  FAIL Package cmd (184ms)


DONE 2 runs, 6 tests, 2 failures in 1.336s

PASS cmd.Test_Flaky2 (re-run 2) (0.00s)
  PASS Package cmd (162ms)

=== Failed
=== FAIL: cmd Test_Flaky2 (0.00s)
    main_test.go:21: Flaky test failed!

=== FAIL: cmd Test_Flaky2 (re-run 1) (0.00s)
    main_test.go:21: Flaky test failed!

DONE 6 runs, 10 tests, 5 failures in 2.546s

I need a way to extract only the === Failed section, if it exists, so that I can loop over the failed tests after this section. This approach would allow me to avoid iterating over all test cases for every package and would help identify flaky tests more quickly, especially when working on a large project with many test cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant