Loss of retry telemetry due to breaking API changes in Cypress 13

[ramosbugs]

Current behavior

#27230 introduced significant breaking API changes in Cypress 13 related to the module API and event handlers. An assumption was made that these fields are "not useful for end users" (quote from the PR description), but it's now effectively impossible for Cypress plugins to collect information about test retries. The recently-closed issue #27390 suggests that other users may have been relying on those fields in prior versions of Cypress as well.

In trying to make my own plugin compatible with Cypress 13, I've noticed the following issues related to the API changes:

1. Per-attempt startedAt, endedAt, and duration information is no longer available. Plugins can no longer report this level of detail to end users, even though it is useful for tracking performance regressions, flaky tests, etc. Instead, the TestResult includes a single duration field. In the case of retried tests, it's not obvious from the API whether this is the duration of the first attempt, last attempt, an average of all attempts, or a sum of all attempts:

   cypress/cli/types/cypress-npm-api.d.ts

   Line 170 in 0f11f87

   duration: number

2. The runnable id is no longer reported for Mocha events. While this field feels more internal than the timing information above, it was previously the only apparent way to unambiguously identify the test an event referred to. For example, the payload of Mocha Runner.constants.EVENT_TEST_RETRY (retry) events includes a title field, but it doesn't have a titlePath function or include a parent field necessary to compute the full title path. It's entirely possible for distinct tests within a Cypress spec to have the same title but lie within different describe() scopes, and the event payload no longer allows plugins to figure out which of those tests is relevant to each event. Previously, plugins could build a map from titlePath() to id by using the complete runnables passed to the Runner.constants.EVENT_SUITE_BEGIN (suite) Mocha event, and then use the id passed to later events such as EVENT_TEST_RETRY to look up the relevant test. See:

   cypress/packages/server/lib/reporter.js

   Line 355 in 0f11f87

   args[0] = _.isPlainObject(args[0]) ? _.omit(args[0], ['id', 'hookId']) : args[0]

There may be other incompatibilities I haven't run into yet because I'm blocked from making further progress on supporting Cypress 13 due to the issues above (particularly the second issue).

Desired behavior

For issue (1), a straightforward solution is to add per-attempt startedAt, endedAt, and duration fields, which would enable a wide variety of plugin/end-user functionality without adding significant complexity to the API. This would also be a backward-compatible change with the current Cypress 13 API.

For issue (2), an ideal solution would be for all of the runnables passed to Mocha event handlers to be full objects that include functions such as titlePath(). It's confusing for some event payloads to have these functions but for others to be plain serializable objects. Since the spirit of #27230 was to improve API consistency, this feels like a move in the same direction, and likely backward compatible.

Test code to reproduce

.

Cypress Version

13.1.0

Node version

all

Operating System

all

Debug Logs

No response

Other

cc: @jennifer-shehane @chrisbreiding

[ramosbugs]

Update: issue (2) can mostly be worked around by listening to EVENT_SUITE_BEGIN and EVENT_SUITE_END events to track the current title path. This isn't ideal and I'm not sure if it handles all test failure cases, but at least it seems possible.

Dealing with before/after hook failures is also difficult without the id field since Mocha changes the test name to include the name of the hook that failed: https://github.com/mochajs/mocha/blob/0be3f78491bbbcdc4dcea660ee7bfd557a225d9c/lib/runner.js#L331. With the test id, we could previously figure out the original name of the test, but now it requires some extra hacks I'm also not confident will handle all cases correctly.

[brandonb927]

Just chiming in here to say that we got bit by this change too after initially missing it in the Release Notes for v13. Our use case is uploading test spec data to Xray after every spec executes using the after:spec event. We lost per-test retry and timing data in this upgrade.

[bixelbg]

This issue affects me too, i have a custom reporter that sends failed tests to my company's private channels, since 13, all the results dissapeard. Somewhat of a fix is to enable video capturing in the config file that was disabled in the 13th version.

[aiotcms]

We have a plugin which reports results to AIO Tests, a test management tool in Jira, which used to rely on the testId values and their correlation in the screenshots object. The testid field has been completely removed, causing loss of important correlations. Also, the attempts object used to provide important information on failures and timings, which too has been trimmed down to just the status. Can these be please reverted.

[aiotcms]

Can the developers please update on this. Basically any kind of external reporting will break in the absence of this data. Are there any workarounds to this? How can plugins be developed with lack of such information? Documentation of these specs - are they valid anymore? https://docs.cypress.io/api/plugins/after-spec-api#Delete-the-recorded-video-if-no-tests-retried

[RicardoVaranda]

@jennifer-shehane Any update on this? Similar to @aiotcms we are reliant on this data as we can't leverage external cloud providers like Cypress Cloud in order to run our tests. It really seems like you have limited the telemetry for anyone not using that solution with no real work around?

[alina-jakubowicz]

@jennifer-shehane Are there any updates on this? Due to this issue, we can't integrate AIO Tests tool with our tests.

[mehulparmariitr]

We have a cypress listener which reads attempts[index].startedAt to get the Test start time and reads attempts[i].duration to get the Test Duration. Now how to get this information from report? @ramosbugs

Note I am talking about individual Testcase and not startTime and duration of the Test Execution. @jennifer-shehane