Cypress doesn't exit if a test fails in Gitlab-ci

[Elliott-Rose-BSC]

Current behavior

Hello, this is releated to issue #5845

I have a pipeline that runs cypress tests with the browser set to Chromium. The tests exit normally if all tests pass, but if there is a failure, no exit code is produced and Cypress just hangs. For now, I have a timeout set so that it will kill the job, but obviously this is undesirable. I have looked through the issues, but I haven't found any good solutions. I cannot use the start-server-and-test plugin as my server is Python based.

Desired behavior

Cypress should exit after a test failure in Gitlab-ci.

Test code to reproduce

Here's an abbreviated version of the script. Code to create a dash_server.py file can be found here.

e2e-mobile-test:
  image: cypress/included:cypress-13.11.0-node-20.14.0-chrome-125.0.6422.141-1-ff-126.0.1-edge-125.0.2535.85-1
  stage: test
  timeout: 6 minutes
  allow_failure: true
  variables:   
    GIT_STRATEGY: fetch
  script:   
     - echo "Start application in background"
        - python dash_server.py --cypress-testing True &
        - npm ci
        - npx cypress run --config-file cypress_mobile.config.js --browser chromium
        - echo $?

When the test fails, the exit code is never printed, and the job waits until timeout.

Cypress Version

13.6.2

Node version

20.14.0

Operating System

Ubuntu 22.04

Debug Logs

No response

Other

No response

[MikeMcC399]

@Elliott-Rose-BSC

If Cypress doesn't exit then it looks like it is hanging for some reason.

I'm not sure which version of Cypress you are using, since in your workflow you list cypress/included:cypress-13.11.0-node-20.14.0-chrome-125.0.6422.141-1-ff-126.0.1-edge-125.0.2535.85-1, however lower down you list Cypress 13.6.2

Running

  image:
    name: cypress/included:13.11.0
    entrypoint: [""]

in GitLab with one deliberately failing test and one passing test exits as follows:

  (Run Finished)
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  show-fail.cy.js                          127ms        1        -        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  spec.cy.js                               00:02        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 2 failed (50%)                      00:02        2        1        1        -        -  
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

So my suggestion would be to try to find out where it is hanging, and then why it is hanging.

I don't believe that your - echo $? will execute if Cypress returns a failure - see https://docs.gitlab.com/ee/ci/yaml/script.html#ignore-non-zero-exit-codes

Also I wonder how your workflow can execute without overriding the entrypoint https://github.com/cypress-io/cypress-docker-images/tree/master/included#gitlab-pipelines.

[MikeMcC399]

I wonder too how you can test using --browser chromium when there is no Chromium browser in the Cypress Docker image:

cypress/included:cypress-13.11.0-node-20.14.0-chrome-125.0.6422.141-1-ff-126.0.1-edge-125.0.2535.85-1

I would expect you to see the error message

Browser: chromium was not found on your system or is not supported by Cypress.

[Elliott-Rose-BSC]

I wonder too how you can test using --browser chromium when there is no Chromium browser in the Cypress Docker image:

cypress/included:cypress-13.11.0-node-20.14.0-chrome-125.0.6422.141-1-ff-126.0.1-edge-125.0.2535.85-1

I would expect you to see the error message

Browser: chromium was not found on your system or is not supported by Cypress.

In fact, if I try to use chrome, I receive an error message saying something like Chrome is not available.

[Elliott-Rose-BSC]

I don't believe that your - echo $? will execute if Cypress returns a failure - see https://docs.gitlab.com/ee/ci/yaml/script.html#ignore-non-zero-exit-codes

Also I wonder how your workflow can execute without overriding the entrypoint https://github.com/cypress-io/cypress-docker-images/tree/master/included#gitlab-pipelines.

Ah, great. These look like a good place to get started. I'll give these a shot, thanks!

[MikeMcC399]

@Elliott-Rose-BSC

In fact, if I try to use chrome, I receive an error message saying something like Chrome is not available.

That doesn't fit in with the previous information as cypress/included:cypress-13.11.0-node-20.14.0-chrome-125.0.6422.141-1-ff-126.0.1-edge-125.0.2535.85-1 definitely includes Chrome. Something else must be going wrong if you're getting a message that Chrome isn't available.

[Elliott-Rose-BSC]

Yes, you're right. In fact I didn't have the executor set to Docker, so this is why it was using the local chromium install. From what I've read, I can't mix shell commands in a job that runs with the docker executor, so for now, I've removed the image and I'm running against the local chromium install, which has been updated to version 130.0.6723.91. I've also added the catch for the non-zero exit code and unfortunately, I'm still running into the same issue. I am seeing a new issue, but I'm not sure if this is related. I still get this error after explicitly setting the TERM in the job to xterm-265color.