Transient CI HTTP 502 Bad Gateway errors

[MikeMcC399]

Issue

The CI workflow circle.yml often experiences HTTP 502 Bad Gateway errors causing CI to report an overall failure.

Rerunning the workflow once or twice will usually recover. (Without write access to CircleCI it is not possible to re-run individual failed tests. I work around this by pushing a dummy amend commit which causes the whole workflow to be re-run.)

CI needs to be more reliable.

Background

• The CI workflow circle.yml runs the standard set of Cypress scaffolded E2E tests located in factory/test-project/cypress/e2e.
• The tests are hard-coded to access https://example.cypress.io and https://jsonplaceholder.cypress.io
• https://example.cypress.io and https://jsonplaceholder.cypress.io are hosted on cloudflare
• Tests are run on CircleCI-hosted runners in the CircleCI network

Logs

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/2150/workflows/aca1a862-49bb-407b-90e9-4a3bb7b261a3/jobs/65630

Running:  2-advanced-examples/waiting.cy.js                                             (19 of 20)


  Waiting
    ✓ cy.wait() - wait for a specific amount of time (4447ms)
    1) cy.wait() - wait for a specific route


  1 passing (11s)
  1 failing

  1) Waiting
       cy.wait() - wait for a specific route:

      Timed out retrying after 4000ms
      + expected - actual

      -502
      +[ 200, 304 ]

      at Context.eval (webpack://test-project/./cypress/e2e/2-advanced-examples/waiting.cy.js:28:54)

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/2149/workflows/787f899c-1ca2-4c35-86bf-507b8a4731bb/jobs/65615

  Running:  2-advanced-examples/network_requests.cy.js                                    (12 of 20)


  Network Requests
    ✓ cy.request() - make an XHR request (448ms)
    1) cy.request() - verify response using BDD syntax
    ✓ cy.request() with query parameters (177ms)
    ✓ cy.request() - pass result to the second request (257ms)
    ✓ cy.request() - save response in the shared test context (243ms)
    ✓ cy.intercept() - route responses to matching requests (955ms)


  5 passing (5s)
  1 failing

  1) Network Requests
       cy.request() - verify response using BDD syntax:

      AssertionError: expected 502 to be one of [ 500, 501 ]
      + expected - actual

      -502
      +[ 500, 501 ]

      at Context.eval (webpack://test-project/./cypress/e2e/2-advanced-examples/network_requests.cy.js:28:74)

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/2134/workflows/9c75a08b-071b-4408-a634-08ef2bd258a6/jobs/65125

  Running:  2-advanced-examples/utilities.cy.js                                           (17 of 20)


  Utilities
    1) Cypress._ - call a lodash method
    ✓ Cypress.$ - call a jQuery method (314ms)
    ✓ Cypress.Blob - blob utilities and base64 string conversion (295ms)
    ✓ Cypress.minimatch - test out glob patterns against strings (170ms)
    ✓ Cypress.Promise - instantiate a bluebird promise (1161ms)


  4 passing (5s)
  1 failing

  1) Utilities
       Cypress._ - call a lodash method:
     CypressError: `cy.request()` failed on:

https://jsonplaceholder.cypress.io/users

The response we received from your web server was:

  > 502: Bad Gateway

This was considered a failure because the status code was not `2xx` or `3xx`.

If you do not want status codes to cause failures pass the option: `failOnStatusCode: false`

-----------------------------------------------------------

The request we sent was:

Method: GET
URL: https://jsonplaceholder.cypress.io/users
Headers: {
  "Connection": "keep-alive",
  "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/13.13.0 Chrome/118.0.5993.159 Electron/27.3.10 Safari/537.36",
  "accept": "*/*",
  "accept-encoding": "gzip, deflate"
}

-----------------------------------------------------------

The response we got was:

Status: 502 - Bad Gateway
Headers: {
  "date": "Wed, 10 Jul 2024 13:59:00 GMT",
  "content-type": "text/html; charset=UTF-8",
  "transfer-encoding": "chunked",
  "connection": "keep-alive",
  "x-frame-options": "SAMEORIGIN",
  "referrer-policy": "same-origin",
  "cache-control": "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0",
  "expires": "Thu, 01 Jan 1970 00:00:01 GMT",
  "cf-ray": "8a111261abce576d-IAD",
  "server": "cloudflare"
}
Body: <!DOCTYPE html>
<html lang="en">
  <head>
    <title>Cypress.io - Error</title>
    <style>
        /*! normalize.css v8.0.1 | MIT License | github.com/necolas/normalize.css */html{line-height:1.15}main{display:block}b{font-weight:bolder}img{border-style:none}[type=button],[type=reset],[type=submit]{-webkit-appearance:button}[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner{border-style:none;padding:0}[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring{outline:1px dotted ButtonText}[type=checkbox],[type=radio]{box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details{display:block}@keyframes vjs-spinner-show{to{visibility:visible}}@-webkit-keyframes vjs-spinner-show{to{visibility:visible}}@keyframes vjs-spinner-spin{to{transform:rotate(1turn)}}@-webkit-keyframes vjs-spinner-spin{to{-webkit-transform:rotate(1turn)}}@keyframes vjs-spinner-fade{0%{border-top-color:#73859f}20%{border-top-color:#73859f}35%{border-top-color:#fff}60%{border-top-color:#73859f}to{border-top-color:#73859f}}@-webkit-keyframes vjs-spinner-fade{0%{border-top-color:#73859f}20%{border-top-color:#73859f}35%{border-top-color:#fff}60%{border-top-color:#73859f}to{border-top-color:#73859f}}@supports (display:grid){.vjs-text-track-settings .vjs-modal-dialog-content{display:grid;grid-template-columns:1fr 1fr;grid-template-rows:1fr auto}.vjs-text-track-settings .vjs-track-settings-colors{display:block;grid-column:1;grid-row:1}.vjs-text-track-settings .vjs-track-settings-font{grid-column:2;grid-row:1}.vjs-text-track-settings .vjs-track-settings-controls{grid-column:2;grid-row:2}}
/*! PhotoSwipe main CSS by Dmitry Semenov | photoswipe.com | MIT license */
/*! ...


https://on.cypress.io/request
      at <unknown> (https://example.cypress.io/__cypress/runner/cypress_runner.js:132776:72)
      at tryCatcher (https://example.cypress.io/__cypress/runner/cypress_runner.js:1807:23)
      at Promise._settlePromiseFromHandler (https://example.cypress.io/__cypress/runner/cypress_runner.js:1519:31)
      at Promise._settlePromise (https://example.cypress.io/__cypress/runner/cypress_runner.js:1576:18)
      at Promise._settlePromise0 (https://example.cypress.io/__cypress/runner/cypress_runner.js:1621:10)
      at Promise._settlePromises (https://example.cypress.io/__cypress/runner/cypress_runner.js:1701:18)
      at _drainQueueStep (https://example.cypress.io/__cypress/runner/cypress_runner.js:2407:12)
      at _drainQueue (https://example.cypress.io/__cypress/runner/cypress_runner.js:2400:9)
      at Async._drainQueues (https://example.cypress.io/__cypress/runner/cypress_runner.js:2416:5)
      at Async.drainQueues (https://example.cypress.io/__cypress/runner/cypress_runner.js:2286:14)
  From Your Spec Code:
      at Context.eval (webpack://test-project/./cypress/e2e/2-advanced-examples/utilities.cy.js:10:7)

[MikeMcC399]

Closing, as this issue appears not to be happening at the moment, although nothing I know about has changed to make it go away.

[MikeMcC399]

Still occurring

[jennifer-shehane]

@MikeMcC399 Not sure why these were happening before, is this still occuring?

[MikeMcC399]

@jennifer-shehane

Not sure why these were happening before, is this still occuring?

It's a sporadic issue which keeps occurring. It could just be Cloudflare overload.

[ibcht]

I encountered the same issue (#1276). 4 attempts yet, most of circle CI tests have failed. It's either the tests which are failing (HTTP 504) or even the Docker build, during the npm download phase (npm error errno ECONNRESET). From one attempt to another, the tests are failing randomly, it's not always the same.

[MikeMcC399]

@ibcht

I encountered the same issue (#1276). 4 attempts yet, most of circle CI tests have failed. It's either the tests which are failing (HTTP 504) or even the Docker build, during the npm download phase (npm error errno ECONNRESET). From one attempt to another, the tests are failing randomly, it's not always the same.

The network seems to be working better today. I was able to get a successful CI run.