feat: implement experimental retries

[AtofStryker]

• Closes N/A

Additional details

adds the implementation to support experimentalStrategy and experimentalOptions for the experimental retries configuration, which is to be used with test burn in. This adds two experimental strategies the user can leverage:

• detect-flake-and-pass-on-threshold
• detect-flake-but-always-fail

Each will have an experimentalOptions that can provide a maxRetries

The first option, detect-flake-and-pass-on-threshold, has a passesRequired option that means the test needs to pass a certain amount of times before marking the test as passed. If this cannot happen, the test is marked as failed

The second option, detect-flake-but-always-fail, has a stopIfAnyPassed option. By default, detect-flake-but-always-fail will run every retry in maxRetries if the test initially fails. This will short circuit if stopIfAnyPassed is true and a retry passes. Regardless of the retries test status (pass or fail), the test will always fail (assuming it fails on the first try)

current implementation (GA)

retries: 2

Experimental options

Detect Flake and Pass on Threshold

experimentalStrategy: 'detect-flake-and-pass-on-threshold'
experimentalOptions: {
  maxRetries: 9,
  passesRequired: 5
}

Detect Flake but Always Fail

experimentalStrategy: 'detect-flake-but-always-fail'
experimentalOptions: {
  maxRetries: 5,
  stopIfAnyPassed: false
}

Detect Flake but Always Fail (stopIfAnyPassed=true)

experimentalStrategy: 'detect-flake-but-always-fail'
experimentalOptions: {
  maxRetries: 10,
  stopIfAnyPassed: true
}

Steps to test

Most of this PR is tests added to verify the behavior works correctly, so there are multiple ways to test this:

• unit-test that runs as a cypress driver test to verify behavior of calculateTestStatus. This function is subject to change in the future and adding/editing tests in the driver to verify its behavior is likely the easiest way to go to verify changes

• cy-in-cy tests, retries.experimentalRetries.mochaEvents.cy.ts and runner.experimentalRetries.mochaEvents.cy.ts inside the @packages/app are added for 3 sample projects that change the test status / attempts run. These are

  • detect-flake-and-pass-on-threshold with a maxRetries of 9 (10 attempts total, if applicable) and a passesRequired of 5
  • detect-flake-but-always-fail with a maxRetries of 9 (10 attempts total) and stopIfAnyPassed is false
  • detect-flake-but-always-fail with a maxRetries of 9 (10 attempts total, if applicable) and stopIfAnyPassed is true

  These projects run against tests in runner.experimentalRetries.mochaEvents.cy.ts in order to guarantee (see code comments to explain strategy outcomes inside test file):

  • before|beforeAll hooks that fail skip the test suite.
  • beforeEach hooks that fail on the max attempts allowed by the current test strategy still skip the test suite.
  • afterEach hooks that fail on the max attempts allowed by the current test strategy still skip the test suite.
  • after|afterAll hooks that fail skip the test suite.
  • verify the correct retry logic is applied on failures with only and no retry logic on initial passed test with only
  • passed tests do not retry

  Additionally, and more importantly, these projects run against tests in retries.experimentalRetries.mochaEvents.cy.ts in order to guarantee (see code comments to explain strategy outcomes inside test file):

  • expected number of attempts are executed based on experimentalStrategy and the correct test status is applied in the run.
  • number of attempts with pass/fail status and overall outer status captured in the mocha snapshots. I'd like to find a better way to verify/test this, but this seems like an acceptable current avenue to accomplish this

• system-tests called experimental_retries to verify the output of the mocha reporter has been added to make sure correct pass/fail is reported for a given test strategy and the correct information is printed to the console. These run in chrome, firefox, and electron currently.

What eventually needs to be tested

• test outer status sent to the cloud inclusive of all attempts for burn in to make sure the payload looks correct. Right now our source of truth for this is the lib/reporter.js in @package/server
• specific UI testing to verify attempt pass/fail. This is handled currently in the cy-in-cy tests but mocha snapshots are extremely difficult to comb through and understand what went wrong / why.

How has the user experience changed?

Users will now be able to provide experimentalStrategy and experimentalOptions in their global config and watch the new form of retries take hold in their test

What is NOT in this PR

TestConfigOverrides

Currently, testConfigOverrides for retries on the test level is not tested or currently supported with the experimental retry options. The goal eventually will be to opt-out users from the experiment if the GA retries configuration is provided in the test, like this:

it('test', { retries: 1 }, () => // opt-out of experimental config and use regular test retries)

Refactors to implementation

The implementation of the experimental retries options lives in 8a4526a. There is likely code here that can be cleaned up and simplified because the current implementation is some what invasive as we are

• patching the mocha package in the driver to support retries on passes, though the patch is small
• handling exception cases in runner.ts to make sure retries are stopped/queued appropriately
• adding a method on the test runnable itself, called calculateTestStatus inside src/cypress/mocha which returns an object to the mocha package to determine whether or not to requeue the test
• a new field, called _cypressTestStatusInfo, which is used by both the cypress reporter in console and in the browser, in order to correctly display the outer test status.

Know bugs

Currently there is an issue with the server console reporter reporting the test status too early before an afterEach hook has had a chance to run and report pass/fail. This likely has to do with us emitting the pass/fail event too early to the server & on the retry itself, meaning we need to emit that later once we have a more full picture.

Here is an example of this bug, where the attempt is logged twice, when it should really be printed once, and printing the status twice, which changes the expected status code in the system test

This issue should be resolved in 072805c

Risk

Likely the biggest thing to address here is possible risk, as the new implementation is applied globally throughout cypress, whether or not it is used. Currently, if no strategy is provided, we treat the config as detect-flake-and-pass-on-threshold up to the retry limit with passesRequired being 1. There is a lot being touched here, but if we are able to get a green build without altering the exterior implementation, then I think we can confidently say this is non breaking (which it should not be breaking)

Reviewing

This is a large change, where the implementation changes are massive but can have a massive downstream impact to the code. To make this easier to review, the commits are broken up as follows:

• implementation 8a4526a
• updates to existing snapshots/tests to ensure backwards capabilities 70d0172
• added cy-in-cy tests to guarantee new functionality 89426e3
• added unit tests to exercise calculateTestStatus function c69d58b
• added system tests to verify reporter output 9abc367
• fixed bugs with aftereach console output into server reporter 072805c
• make sure hook failures print correct last attempt status 813c9ce

PR Tasks

• Have tests been added/updated?
• Has a PR for user-facing changes been opened in cypress-documentation? This will be handled as a separate work item
• Have API changes been updated in the type definitions? (N/A, the experimental options have already been added in feat: add new experimental retries configuration #27412)

[cypress]

1 flaky test on run #51314 ↗︎

0

217

21

0

1

Details:

chore: improve types within calculateTestStatus inside mocha.ts
Project: cypress	Commit: 6343ae9c30
Status: Passed	Duration: 09:32 💡
Started: Sep 27, 2023 6:14 PM	Ended: Sep 27, 2023 6:23 PM

  cypress/e2e/project-setup.cy.ts • 1 flaky test • launchpad-e2e

View Output Video

Test		Artifacts
... > can skip setup CT testing for an E2E project		Test Replay Output Screenshots

Review all test suite changes for PR #27826 ↗︎

[AtofStryker]

@chrisbreiding I was able to rebase the PR with comment updates to clarify test vs test attempt. Let me know if something can be clearer or is confusing!

[AtofStryker]

@chrisbreiding @cacieprins I think I fixed the afterEach bug mentioned in the PR description in 072805c. If we are in a case where the outer status has failed but the test attempt passed, we want to delay signaling that the test passed because a hook might fail someplace in the suite. snapshots are all updated and look good, though I still think the exit code should be an 8 and not a 9 on the system tests, but this looks to be another issue

[AtofStryker]

@chrisbreiding @cacieprins I pushed a fixed up in 89a21a9 that fixes the server reporter to correctly print attempts on hook failures. The system tests take the runner ui cy-in-cy tests and run the m in a system test to verify output. There are some bugs in the system test snapshot, like NaNms in the output, and some of the semantics need to be updated, such as the server reporter assuming the runnable is a test (it could be a hook as well), but think we should hold off on including those updates here.