Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: failing Create Instance api call no longer skips spec #30697

Merged
merged 7 commits into from
Dec 3, 2024
Merged

fix: failing Create Instance api call no longer skips spec #30697

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6386b9e
do not retry instance creation, report that cannot proceed with faile…
cacieprins Dec 2, 2024
f01beae
update system tests
cacieprins Dec 2, 2024
6874bf2
changelog
cacieprins Dec 2, 2024
563f913
changelog
cacieprins Dec 2, 2024
821a90b
Update cli/CHANGELOG.md
cacieprins Dec 2, 2024
97694a8
update error language
cacieprins Dec 2, 2024
48de0bb
snapshots
cacieprins Dec 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cli/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
<!-- See the ../guides/writing-the-cypress-changelog.md for details on writing the changelog. -->

## 13.16.1

_Released 11/26/2024 (PENDING)_

**Bugfixes:**

- Support multiple imports of one module with multiple lines. Addressed in [#30314](https://github.com/cypress-io/cypress/pull/30314).
- During record or parallel runs, execution will fail if Cypress is unable to confirm the creation of an instance instead of skipping the spec. Addresses [#30628](https://github.com/cypress-io/cypress/issues/30628).
cacieprins marked this conversation as resolved.
Show resolved Hide resolved

## 13.16.0

Expand Down
4 changes: 3 additions & 1 deletion packages/errors/__snapshot-html__/CLOUD_CANNOT_PROCEED_IN_SERIAL.html
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions packages/errors/src/errors.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ export const AllCypressErrors = {

${fmt.highlightSecondary(arg1.response)}

Because you passed the ${fmt.flag(`--record`)} flag, this run cannot proceed because it requires a valid response from our servers.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just removing the double 'because'

Suggested change
Because you passed the ${fmt.flag(`--record`)} flag, this run cannot proceed because it requires a valid response from our servers.
Because you passed the ${fmt.flag(`--record`)} flag, this run cannot proceed since it requires a valid response from our servers.


${fmt.listFlags(arg1.flags, {
group: '--group',
ciBuildId: '--ciBuildId',
Expand Down
28 changes: 13 additions & 15 deletions packages/server/lib/cloud/api/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,22 +458,20 @@ export default {
'platform',
])

return retryWithBackoff((attemptIndex) => {
return rp.post({
body,
url: recordRoutes.instances(runId),
json: true,
encrypt: preflightResult.encrypt,
timeout: timeout ?? SIXTY_SECONDS,
headers: {
'x-route-version': '5',
'x-cypress-run-id': runId,
'x-cypress-request-attempt': attemptIndex,
},
})
.catch(RequestErrors.StatusCodeError, formatResponseBody)
.catch(tagError)
return rp.post({
body,
url: recordRoutes.instances(runId),
json: true,
encrypt: preflightResult.encrypt,
timeout: timeout ?? SIXTY_SECONDS,
headers: {
'x-route-version': '5',
'x-cypress-run-id': runId,
'x-cypress-request-attempt': 0,
},
})
.catch(RequestErrors.StatusCodeError, formatResponseBody)
.catch(tagError)
},

postInstanceTests (options) {
Expand Down
204 changes: 97 additions & 107 deletions system-tests/__snapshots__/record_spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1124,13 +1124,17 @@ We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

Because you passed the --record flag, this run cannot proceed because it requires a valid response from our servers.

`

exports['e2e record api interaction errors create run 500 when grouping without parallelization errors and exits 1'] = `
We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

Because you passed the --record flag, this run cannot proceed because it requires a valid response from our servers.

The --group flag you passed was: foo
The --ciBuildId flag you passed was: ciBuildId123

Expand Down Expand Up @@ -1193,6 +1197,8 @@ We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

Because you passed the --record flag, this run cannot proceed because it requires a valid response from our servers.

`

exports['e2e record api interaction errors update instance 500 does not proceed and exits with error when parallelizing and updating instance 1'] = `
Expand Down Expand Up @@ -1409,6 +1415,8 @@ We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

Because you passed the --record flag, this run cannot proceed because it requires a valid response from our servers.

`

exports['e2e record api interaction errors postInstanceTests without parallelization errors and exits 1'] = `
Expand All @@ -1435,6 +1443,8 @@ We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

Because you passed the --record flag, this run cannot proceed because it requires a valid response from our servers.

The --group flag you passed was: foo
The --ciBuildId flag you passed was: 1

Expand Down Expand Up @@ -1526,6 +1536,8 @@ We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

Because you passed the --record flag, this run cannot proceed because it requires a valid response from our servers.

`

exports['e2e record api interaction errors update instance stdout warns but proceeds 1'] = `
Expand Down Expand Up @@ -1705,113 +1717,6 @@ exports['e2e record api interaction errors uploading assets warns but proceeds 1
Recorded Run: https://dashboard.cypress.io/projects/cjvoj7/runs/12


`

exports['e2e record api interaction errors api retries on error warns and does not create or update instances 1'] = `
We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 3 more times in X second(s)...

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 2 more times in X second(s)...

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 1 more time in X second(s)...


====================================================================================================

(Run Starting)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (record_pass.cy.js) │
│ Searched: cypress/e2e/record_pass* │
│ Params: Tag: nightly, Group: foo, Parallel: true │
│ Run URL: https://dashboard.cypress.io/projects/cjvoj7/runs/12 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 3 more times in X second(s)...


────────────────────────────────────────────────────────────────────────────────────────────────────

Running: record_pass.cy.js (1 of 1)
Estimated: X second(s)


record pass
✓ passes
- is pending


1 passing
1 pending


(Results)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 1 │
│ Skipped: 0 │
│ Screenshots: 1 │
│ Video: false │
│ Duration: X seconds │
│ Estimated: X second(s) │
│ Spec Ran: record_pass.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘


(Screenshots)

- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)


(Uploading Cloud Artifacts)

- Video - Nothing to upload
- Screenshot - 1 kB /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Nothing to upload - Test Replay is disabled for this project. Enable Test Replay in Cloud project settings

Uploading Cloud Artifacts: . . . . .

(Uploaded Cloud Artifacts)

- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/1 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png

====================================================================================================

(Run Finished)


Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ record_pass.cy.js XX:XX 2 1 - 1 - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 2 1 - 1 -


───────────────────────────────────────────────────────────────────────────────────────────────────────

Recorded Run: https://dashboard.cypress.io/projects/cjvoj7/runs/12


`

exports['e2e record api interaction errors sendPreflight [F1] socket errors fails after retrying 1'] = `
Expand Down Expand Up @@ -4479,3 +4384,88 @@ http://localhost:1234/capture-protocol/upload/?x-amz-credential=XXXXXXXX&x-amz-s


`

exports['e2e record api interaction errors api retries on error errors and does not create or update instances when parallel 1'] = `
We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 3 more times in X second(s)...

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 2 more times in X second(s)...

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 1 more time in X second(s)...


====================================================================================================

(Run Starting)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (record_pass.cy.js) │
│ Searched: cypress/e2e/record_pass* │
│ Params: Tag: nightly, Group: foo, Parallel: true │
│ Run URL: https://dashboard.cypress.io/projects/cjvoj7/runs/12 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

Because you passed the --parallel flag, this run cannot proceed because it requires a valid response from our servers.

The --group flag you passed was: foo
The --ciBuildId flag you passed was: ciBuildId123

`

exports['e2e record api interaction errors api retries on error errors and does not create or update instances when recording and not parallel 1'] = `
We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 3 more times in X second(s)...

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 2 more times in X second(s)...

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

We will retry 1 more time in X second(s)...


====================================================================================================

(Run Starting)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (record_pass.cy.js) │
│ Searched: cypress/e2e/record_pass* │
│ Params: Tag: nightly, Group: false, Parallel: false │
│ Run URL: https://dashboard.cypress.io/projects/cjvoj7/runs/12 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

We encountered an unexpected error communicating with our servers.

StatusCodeError: 500 - "Internal Server Error"

Because you passed the --record flag, this run cannot proceed because it requires a valid response from our servers.

`
37 changes: 30 additions & 7 deletions system-tests/test/record_spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1615,7 +1615,11 @@ describe('e2e record', () => {

setupStubbedServer(routes)

it('warns and does not create or update instances', function () {
beforeEach(() => {
count = 0
})

it('errors and does not create or update instances when parallel', function () {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason we don't make these async/await tests to help with readability?

process.env.API_RETRY_INTERVALS = '1000,2000,3000'

return systemTests.exec(this, {
Expand All @@ -1628,6 +1632,7 @@ describe('e2e record', () => {
parallel: true,
snapshot: true,
ciBuildId: 'ciBuildId123',
expectedExitCode: 1,
})
.then(() => {
const urls = getRequestUrls()
Expand All @@ -1638,12 +1643,30 @@ describe('e2e record', () => {
'POST /runs',
'POST /runs',
'POST /runs/00748421-e035-4a3d-8604-8468cc48bdb5/instances',
'POST /runs/00748421-e035-4a3d-8604-8468cc48bdb5/instances',
'POST /instances/e9e81b5e-cc58-4026-b2ff-8ae3161435a6/tests',
'POST /instances/e9e81b5e-cc58-4026-b2ff-8ae3161435a6/results',
'PUT /screenshots/1.png',
'PUT /instances/e9e81b5e-cc58-4026-b2ff-8ae3161435a6/artifacts',
'PUT /instances/e9e81b5e-cc58-4026-b2ff-8ae3161435a6/stdout',
])
})
})

it('errors and does not create or update instances when recording and not parallel', function () {
process.env.API_RETRY_INTERVALS = '1000,2000,3000'

return systemTests.exec(this, {
key: 'f858a2bc-b469-4e48-be67-0876339ee7e1',
configFile: 'cypress-with-project-id.config.js',
spec: 'record_pass*',
tag: 'nightly',
record: true,
snapshot: true,
expectedExitCode: 1,
})
.then(() => {
const urls = getRequestUrls()

expect(urls).to.deep.eq([
'POST /runs',
'POST /runs',
'POST /runs',
'POST /runs',
'POST /runs/00748421-e035-4a3d-8604-8468cc48bdb5/instances',
])
})
Expand Down
Loading