feat: more informative error messages when Test Replay fails to capture #29312
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cacieprins
changed the base branch from
develop
to
cacie/refactor/artifact-upload-to-ts
There was a problem hiding this comment.
I like the error message text that will be displayed. I wish they could be a little more actionable, but I don't quite see how they can be.
If we had any expectations of wanting to change the actions we recommend to people in this situations in the future, I would recommend putting an on.cypress.io link to our error-messages page so that we could more easily update the actions. But I'm not sure this is the case with these situations.
cacieprins
changed the title
Base automatically changed from
cacie/refactor/artifact-upload-to-ts
to
develop
April 12, 2024 13:20
cacieprins
force-pushed
the
cacie/feat/improved-upload-error-msgs
branch
2 times, most recently
from
7ce569e to
6771fdb
Compare
cacieprins
force-pushed
the
cacie/feat/improved-upload-error-msgs
branch
from
6771fdb to
e5f3cbd
Compare
4 flaky tests on run #55272 ↗︎
Details:
|
|||||||||||||||||||||||||||||||||
| Test | Artifacts | |
|---|---|---|
| scaffolding component testing > create-react-app > scaffolds component testing for Create React App v5 project |
Test Replay
Screenshots
|
|
e2e/origin/config_env.cy.ts • 1 flaky test • 5x-driver-chrome:beta
| Test | Artifacts | |
|---|---|---|
| cy.origin- Cypress.config() > serializable > overwrites different values in secondary if one exists in the primary |
Test Replay
|
|
commands/net_stubbing.cy.ts • 2 flaky tests • 5x-driver-webkit
| Test | Artifacts | |
|---|---|---|
| ... > with `resourceType` > can match a proxied image request by resourceType |
Review all test suite changes for PR #29312 ↗︎ | |
|
|
||
| for (const [key, value] of parsedUrl.searchParams) { | ||
| if (SENSITIVE_KEYS.includes(key)) { | ||
| parsedUrl.searchParams.set(key, 'X'.repeat(value.length)) |
There was a problem hiding this comment.
Do we want to match the value length? Not sure if just the length of the string could give something away?
There was a problem hiding this comment.
these keys are fairly temporary, so I'm not sure if it's even necessary to be honest.
ryanthemanuel
approved these changes
Apr 23, 2024
Co-authored-by: Ryan Manuel <[email protected]>
mschile
reviewed
Apr 26, 2024
| @@ -1,12 +1,16 @@ | |||
| <!-- See the ../guides/writing-the-cypress-changelog.md for details on writing the changelog. --> | |||
| ## 13.8.1 | |||
| ## 13.9.0 | |||
There was a problem hiding this comment.
There should be a new section for 13.9.0 since 13.8.1 has already been released.
There was a problem hiding this comment.
addressed in 122cd50
mschile
approved these changes
Apr 26, 2024
jj497
pushed a commit
to jj497/cypress
that referenced
this pull request
May 5, 2024
…re (cypress-io#29312) * new error messages * errors for initialization and capturing test replay * messaging the case where no protocol instance but also no fatal error * adds suggested remedies to initialization errors * differentiation between network and http errors, initial work on db missing error * improve db not found error * add new error type to schema * rm commented dead code * clean up network error creation in uploadStream * make artifact confirmation error msg more general * test display of put instance artifacts failure * changelog * ensure we are displaying errors even in quiet mode * fix pending changelog * new snapshots for protocol errors * improve aggregate error * resolved html error snapshots * fix check-ts * fix test * sanitize temp dir in CLOUD_PROTOCOL_CAPTURE_FAILURE error msg * changelog * fix double entry of certain network errors, error message prop for network errors * fix url arg for network error * Update packages/server/lib/cloud/artifacts/upload_artifacts.ts Co-authored-by: Ryan Manuel <[email protected]> * rm extra formatting from debug * snapshot whitespace * changelog update * +pending * Update cli/CHANGELOG.md * resolve snapshots? * does moving the stack trace fix whitespace in snapshots in ci? * maybe fixes whitespace on electron now? * fully normalize stack traces * remove full normalization * maybe debug stack normalization * rm stack traces from error messages --------- Co-authored-by: Jennifer Shehane <[email protected]> Co-authored-by: Ryan Manuel <[email protected]>
|
Released in This comment thread has been locked. If you are still experiencing this issue after upgrading to |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Additional details
Many of the errors that can crop up when initializing, capturing, or uploading Test Replay recordings were fairly cryptic. This PR attempts to improve that, with differentiated error messages for:
In addition, Test Replay will cease to display in the Artifact Upload manifest if it fails to initialize or if it encounters a fatal error during recording. We print an error message before the upload section of stdout in this case.
Several of these more verbose error messages include paths and URLs that will more readily identify issues and lead to faster and easier resolution. In certain situations, we can provide actionable advice, like: confirming network configuration, increasing available disk space, and ensuring that temporary directories are accessible.
Steps to test
How has the user experience changed?
The existing upload report will still be displayed, however, the following error messages will also be displayed:
When a request to our API to confirm the status of artifact (video, screenshots, and Test Replay) fails
When Test Replay fails to initialize
When Test Replay encounters a fatal error during recording
When S3 returns an HTTP status code that indicates an error
When the Test Replay recording fails to upload due to a system-level network error
When the Test Replay recording fails to upload after several retries
An aggregate error is displayed with each error that was encountered while uploading. We display a recommendation to verify network configuration when we detect that at least one of these errors is a system-level network error.
PR Tasks
cypress-documentation?type definitions?