Incorrect attempts in after:run hook with retries enabled

[csvtuda]

Current behavior

When retries are enabled and a test is flaky (i.e. fails at the beginning, then passes in a later retry), the results reported in the "after:run" hook contain duplicate attempts.

describe('page', () => {
  it('works', () => {
    cy.visit('https://example.cypress.io');
    // simulate a flaky test that passes on the third attempt
    if (Cypress.currentRetry < 2) {
      expect(true).to.eq(false)
    } else {
      expect(true).to.eq(true)
    }
  })
})

Config file:

const { writeFileSync } = require("node:fs");

module.exports = {
  e2e: {
    setupNodeEvents(on, config) {
      on("after:run", (results) => {
        writeFileSync("results.json", JSON.stringify(results, null, 2));
      });
    },
    retries: 2
  },
}

The results reported:

"tests": [
  {
    "attempts": [
      {
        "state": "failed"
      },
      {
        "state": "failed"
      },
      {
        "state": "failed"
      },
      {
        "state": "failed"
      },
      {
        "state": "passed"
      }
    ],
    "displayError": null,
    "duration": 1513,
    "state": "passed",
    "title": [
      "page",
      "works"
    ]
  }
],

Desired behavior

In this situation, Cypress should only report two failing and one passing attempt.

"tests": [
  {
    "attempts": [
      {
        "state": "failed"
      },
      {
        "state": "failed"
      },
      {
        "state": "passed"
      }
    ],
    "displayError": null,
    "duration": 1489,
    "state": "passed",
    "title": [
      "page",
      "works"
    ]
  }
],

Test code to reproduce

1. clone repository with buggy setup: https://github.com/csvtuda/cypress-attempt-bug
2. npm install
3. npx cypress run
4. inspect results.json

Cypress Version

13.17.0

Node version

v22.11.0

Operating System

Windows 10 Pro 19045.5247

Debug Logs

No response

Other

Interestingly, this does not occur when the test does not contain any Cypress command:

describe('page', () => {
  it('works', () => {
    // no cypress visit, but still flaky
    if (Cypress.currentRetry < 2) {
      expect(true).to.eq(false)
    } else {
      expect(true).to.eq(true)
    }
  })
})

The reported results:

"tests": [
  {
    "attempts": [
      {
        "state": "failed"
      },
      {
        "state": "failed"
      },
      {
        "state": "passed"
      }
    ],
    "displayError": null,
    "duration": 1489,
    "state": "passed",
    "title": [
      "page",
      "works"
    ]
  }
],

---

I hope this is not a duplicate issue, I failed to find anything using GitHub's search.

[jennifer-shehane]

I feel like this might be related to this issue: #2777 Do the double state go away if a baseUrl is set - and that baseUrl is visiting in the cy.visit?

[csvtuda]

Yes, defining a baseUrl and visiting it through / fixes the issue.

[jennifer-shehane]

Yah this is not a great side effect of that. I'll add this case to that existing issue and close this as a duplicate.