feat: add CYPRESS_SKIP_VERIFY option to suppress verification

[joshuajtward]

• Closes Being able to disable cypress verify step #22243

Additional details

The PR adds a flag CYPRESS_SKIP_VERIFY to prevent the verification steps from running. These steps are semi-redundant in CI environments, and regularly timeout leading to false negatives in test suites.

Steps to test

• Delete the Cypress cache on your machine and run Cypress with CYPRESS_SKIP_VERIFY=true in your env
• Confirm that the verification messages are not shown ("Looks like this is your first time running Cypress N.N.N...")

How has the user experience changed?

If a user opts in they will not get the verification on Cypress startup

PR Tasks

• Have tests been added/updated?
• Has a PR for user-facing changes been opened in cypress-documentation?
• Have API changes been updated in the type definitions?

[CLAassistant]

All committers have signed the CLA.

[cypress-app-bot]

• Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.

[MikeMcC399]

changelog does not match code

[MikeMcC399]

@joshuajtward

Additional details

The PR adds a flag CYPRESS_SKIP_VERIFY to prevent the verification steps from running. These steps are semi-redundant in CI environments, and regularly timeout leading to false negatives in test suites.

Did you open an issue about timeouts so this could be investigated? I'm seeing some other problems in CI environments with cypress verify where the check is uncovering some other issues, so I wonder if it is a good idea to suppress this check.

[joshuajtward]

Hi @MikeMcC399! I have not opened an issue about the timeouts, but there has been a lot of discussion in this repo about this issue already (see for example #6082). It has been partially addressed by adding the ability to set a long timeout, but I would like to add the option to suppress the error so that we can:

1. reduce the false negatives in our CI pipelines
2. see if suppressing it raises any more informative logs or errors

[MikeMcC399]

@joshuajtward

• Thank you very much for the reference to issue Cypress verification fails randomly during Docker CI #6082 ! I didn't know about that. I haven't seen this happen recently with Cypress Docker images, but maybe this depends on the environment?

[joshuajtward]

Yes @MikeMcC399, I'm pretty sure something is going awry with the child tasks which interrogate the installed vs expected binaries, but it's pretty hard to narrow it down because it only happens on around 2% of jobs and usually passes on a re-run. I don't think it's a sensible use of resources to put DEBUG=cypress:* in all our CI runs...

For our use cases we don't much care about these integrity checks as we're pretty confident in our CI environment's stability (I think that will be a commonly held sentiment), our jobs are all running on the same container infrastructure, and using the same caches. We don't want to have to implement a retry shell script or other such workarounds, so using a flag to opt out seems like a better way forward for us.

If our use of this flag helps to highlight a root cause or underlying problem I will be happy to raise it as an issue in this repo!

[MikeMcC399]

@joshuajtward

Thanks again for your explanation and background! Which CI provider are you using and which package manager (npm, Yarn or pnpm)?

[joshuajtward]

@MikeMcC399 we're using GitHub Actions with a mix of Linux, Windows, and MacOS, and a mix of yarn and npm across a number of repos

[MikeMcC399]

The new environment variable would need to be documented in the table https://docs.cypress.io/guides/references/advanced-installation#Environment-variables so this would mean that a release branch for 13.14.0 (or later if delayed) would need to be created in the docs repo to prepare for release.

[MikeMcC399]

The documentation aspect needs to be covered.

[joshuajtward]

@MikeMcC399 docs PR is here: cypress-io/cypress-documentation#5889

[joshuajtward]

@MikeMcC399 @jennifer-shehane is there anything else I can help with to get this moving?

[MikeMcC399]

@joshuajtward

is there anything else I can help with to get this moving?

Yes, you will need to resolve the conflicts in the branch. Unfortunately this can happen as changes are committed to the develop branch.

This is mentioned in the submitter's guide https://github.com/cypress-io/cypress/blob/develop/CONTRIBUTING.md#pull-requests

[joshuajtward]

@MikeMcC399 sorry I hadn't seen there were more conflicts. I have updated that now, is there anything else?

[MikeMcC399]

@joshuajtward

sorry I hadn't seen there were more conflicts. I have updated that now

👍🏻

is there anything else?

CI has one failing test, however that may be CircleCI flake. I expect that @jennifer-shehane will be able to provide more concrete feedback as the Cypress.io team is more familiar with the individual tests than I am. They can also re-run individual tests.

Since I'm only an external contributor, I don't have write privileges to CircleCI, to the repo or to the PR approval process. I can just provide comments.

[jennifer-shehane]

@joshuajtward Just awaiting @AtofStryker review at this point

[joshuajtward]

Good shout @AtofStryker, that reads more nicely - I've updated it now

[jennifer-shehane]

@joshuajtward There's a linting error

[joshuajtward]

@AtofStryker I have fixed this up and it should be ready for a re-review, thanks!

[AtofStryker]

@joshuajtward Thank you for getting the changes addressed! One last question on my end. I created a binary of your code changes on add6f09. Are you able to download the binary from the link and verify the changes work as expected?

[joshuajtward]

@joshuajtward Thank you for getting the changes addressed! One last question on my end. I created a binary of your code changes on add6f09. Are you able to download the binary from the link and verify the changes work as expected?

Hi @AtofStryker, I have installed the linux arm64 binary and run a few tests, it appears to work as expected

Without the flag it works as normal:

With the flag the verify steps are skipped:

And if the binary is missing then it fails with an illustrative error:

Please let me know if there's any other testing you would like me to do!

[MikeMcC399]

@joshuajtward

I notice that you're running quite an old version of Yarn (v1.18.0) from 2019. Yarn v1.22.22 is the current stable version (see https://classic.yarnpkg.com/en/docs/install).

• Edit: See related chore(deps): update yarn to >=1.22.17 #30048, but probably not an issue for this PR.

[joshuajtward]

@jennifer-shehane are we all set to merge? 🙏

[cypress-bot]

Released in 13.14.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.14.0, please open a new issue.