Missing run details when running programmatically

[debugbit]

Current behavior

We run cypress via module api and collect the results for internal processing and analyzing error details. After upgrade to cypress 13 we are missing information when running cypress.run and that breaks our automations.

• screenshots don't have test and attempt information
• test are missing details for failed attempt - error details, start, end and duration

Desired behavior

Keep the same output as in previous versions of cypress

Test code to reproduce

any cypress.run

Cypress Version

13.0.0, 13.1.0

Node version

v16.17.1

Operating System

any

Debug Logs

No response

Other

No response

[nagash77]

Hi @debugbit , this sounds possibly related to #27694. We released 13.1.0 yesterday which addressed those problems.

Can you share more about your screenshots, the data you collect and how you collect it?

[debugbit]

@nagash77 Sure, we have a script that collects data about each failed attempt and sends it to our internal database for generating report and analysis. The script uses cypress.run and parses its output.

We need the failed attempt details to analyze the most problematic tests and their root cause. After upgrading to cypress 13 we cannot get this information anymore and that prevents us from upgrading.

Here's the relevant output from 13.1.0 (full output):

"tests": [
      {
        "displayError": "Error: error message\n    at Context.eval (webpack://web/./cypress/e2e/xxx.js:9:12)",
        "duration": 1580,
        "state": "failed",
        "title": ["Retries", "Example retries"],
        "attempts": [
          {
            "state": "failed"
          },
          {
            "state": "failed"
          },
          {
            "state": "failed"
          },
          {
            "state": "failed"
          }
        ]
      }
    ]

Here's an output from 12.17.4 for the same test (full output]

"tests": [
        {
          "title": ["Retries", "Example retries"],
          "state": "failed",
          "body": "function () {\n    throw new Error(\"error message\");\n  }",
          "displayError": "Error: error message\n    at Context.eval (webpack://web/./cypress/e2e/xxx.js:9:12)",
          "attempts": [
            {
              "state": "failed",
              "error": {
                "name": "Error",
                "message": "error message",
                "stack": "    at Context.eval (webpack://web/./cypress/e2e/xxx.js:9:12)",
                "codeFrame": {
                  "line": 9,
                  "column": 13,
                  "originalFile": "cypress/e2e/xxx.js",
                  "relativeFile": "examples/webapp/cypress/e2e/xxx.js",
                  "absoluteFile": "/Users/debugbit/frontend/cypress/e2e/xxx.js",
                  "frame": "   7 |     },\n   8 |     function () {\n>  9 |       throw new Error(\"error message\");\n     |             ^\n  10 |     }\n  11 |   );\n  12 | });",
                  "language": "js"
                }
              },
              "videoTimestamp": 2136,
              "duration": 814,
              "startedAt": "2023-09-01T17:40:07.518Z",
              "screenshots": [
                {
                  "name": null,
                  "takenAt": "2023-09-01T17:40:08.069Z",
                  "path": "/Users/debugbit/frontend/cypress/screenshots/xxx.js/Retries -- Example retries (failed).png",
                  "height": 1440,
                  "width": 2560
                }
              ]
            },
            {
              "state": "failed",
              "error": {
                "name": "Error",
                "message": "error message",
                "stack": "    at Context.eval (webpack://web/./cypress/e2e/xxx.js:9:12)",
                "codeFrame": {
                  "line": 9,
                  "column": 13,
                  "originalFile": "cypress/e2e/xxx.js",
                  "relativeFile": "examples/webapp/cypress/e2e/xxx.js",
                  "absoluteFile": "/Users/debugbit/frontend/cypress/e2e/xxx.js",
                  "frame": "   7 |     },\n   8 |     function () {\n>  9 |       throw new Error(\"error message\");\n     |             ^\n  10 |     }\n  11 |   );\n  12 | });",
                  "language": "js"
                }
              },
              "videoTimestamp": 2968,
              "duration": 295,
              "startedAt": "2023-09-01T17:40:08.350Z",
              "screenshots": [
                {
                  "name": null,
                  "takenAt": "2023-09-01T17:40:08.435Z",
                  "path": "/Users/debugbit/frontend/cypress/screenshots/xxx.js/Retries -- Example retries (failed) (attempt 2).png",
                  "height": 1440,
                  "width": 2560
                }
              ]
            },
            {
              "state": "failed",
              "error": {
                "name": "Error",
                "message": "error message",
                "stack": "    at Context.eval (webpack://web/./cypress/e2e/xxx.js:9:12)",
                "codeFrame": {
                  "line": 9,
                  "column": 13,
                  "originalFile": "cypress/e2e/xxx.js",
                  "relativeFile": "examples/webapp/cypress/e2e/xxx.js",
                  "absoluteFile": "/Users/debugbit/frontend/cypress/e2e/xxx.js",
                  "frame": "   7 |     },\n   8 |     function () {\n>  9 |       throw new Error(\"error message\");\n     |             ^\n  10 |     }\n  11 |   );\n  12 | });",
                  "language": "js"
                }
              },
              "videoTimestamp": 3271,
              "duration": 293,
              "startedAt": "2023-09-01T17:40:08.653Z",
              "screenshots": [
                {
                  "name": null,
                  "takenAt": "2023-09-01T17:40:08.747Z",
                  "path": "/Users/debugbit/frontend/cypress/screenshots/xxx.js/Retries -- Example retries (failed) (attempt 3).png",
                  "height": 1440,
                  "width": 2560
                }
              ]
            },
            {
              "state": "failed",
              "error": {
                "name": "Error",
                "message": "error message",
                "stack": "    at Context.eval (webpack://web/./cypress/e2e/xxx.js:9:12)",
                "codeFrame": {
                  "line": 9,
                  "column": 13,
                  "originalFile": "cypress/e2e/xxx.js",
                  "relativeFile": "examples/webapp/cypress/e2e/xxx.js",
                  "absoluteFile": "/Users/debugbit/frontend/cypress/e2e/xxx.js",
                  "frame": "   7 |     },\n   8 |     function () {\n>  9 |       throw new Error(\"error message\");\n     |             ^\n  10 |     }\n  11 |   );\n  12 | });",
                  "language": "js"
                }
              },
              "videoTimestamp": 3576,
              "duration": 282,
              "startedAt": "2023-09-01T17:40:08.958Z",
              "screenshots": [
                {
                  "name": null,
                  "takenAt": "2023-09-01T17:40:09.050Z",
                  "path": "/Users/debugbit/frontend/cypress/screenshots/xxx.js/Retries -- Example retries (failed) (attempt 4).png",
                  "height": 1440,
                  "width": 2560
                }
              ]
            }
          ]
        }
      ],

As you can see 12.17.4 had much more details for each attempt:

• error details, including error stack, message and code frame
• duration
• startedAt
• screenshots are listed per attempt

The output from 13.1.0 is limited does not have the necessary details

• missing attempt error and timing details
• not possible to find what screenshot belongs to what test and attempt

[brandonb927]

I think this is very similar to #27732 and I'll bring up my comment here as well #27732 (comment)

[debugbit]

@nagash77 would you accept a contribution with a fix? The necessary data is already present, it is literally just filtered out. It seems like a straightforward fix:
https://github.com/cypress-io/cypress/blame/dc282cd7e873addb81b6360f83a3f035fb2686df/packages/server/lib/modes/results.ts#L111

[debugbit]

@nagash77 @MikeMcC399 @chrisbreiding

Sorry to bother you guys, this issue is blocking us and it's not clear if we'll be able to use cypress. We need to decide what to do next. Would appreciate some inputs - are you planning to reenable the removed data?

[MikeMcC399]

@debugbit

I'm sorry to hear of your difficulties, however I'm not involved in this issue. (Also I am only a community member, so my influence is limited.) Good luck!

[jennifer-shehane]

@debugbit Right now adding these fields to our Public APIs (like the Module API and after:run results) is not under consideration.

The data around Cypress’s retry logic is available in the Cypress Cloud if you find value in tracking flaky tests.

[RicardoVaranda]

@jennifer-shehane Coming from a corporate background where we simply can't send data externally to services like Cypress Cloud this is extremely disappointing that you have decided to do this.
Why not expose those fields via some sort of flag that can be set by default in the module API?
This decision is affecting us massively as we can no longer get metadata information on why tests are failing. I would urge you to reconsider this decision as like many others we are now having to consider moving away from Cypress due to level of granularity on why the tests failed is being removed.