Cookies are broken with cy.origin

[denke8]

Current behavior

I'm trying to test a WordPress plugin. The plugin does different things depending on if the domain is recognized or not.

For this reason I'm running the exact same test (except the last assertion) twice, once normally, and once wrapped in cy.origin.
experimentalSessionAndOrigin is set to true

The test without the cy.origin works perfectly.

The one wrapped in cy.origin runs twice (test retry):

• On the first iteration, it is unable to log in to /wp-admin. The browser freezes for a while, then it returns to the login screen without any errors
• On the second try, it manages to log in, and displays the WP dashboard, but the very next page navigation kicks us back to the login screen
  

This is fairly repeatable

Desired behavior

No response

Test code to reproduce

describe('Manual plugin activation', () => {
  describe('w/ random domain', () => {
    beforeEach(() => {
      // This command brings up the appropriate docker containers, and sets WP up with a random domain,
      // which it adds to the hosts file so the browser can access it.
      // It then executes Cypress.config('baseUrl', baseUrl); to prepare for the tests
      // This seem to be working correctly, as the page does show up correctly in response to cy.visit()
      cy.resetTestingEnv({ randomBaseUrl: true })
    })

    it('should display signup CTA when autoconfig fails', () => {
      cy.origin(Cypress.config('baseUrl'), {}, () => {
        // cy.adminAuth()
        cy.visit('/wp-admin')

        // eslint-disable-next-line cypress/no-unnecessary-waiting
        cy.wait(500) // The login page clears/resets the input from js whan loaded, so need to wait

        cy.contains('Username').type(Cypress.env('CYPRESS_WP_ADMIN_USER'))
        cy.contains('Password').type(Cypress.env('CYPRESS_WP_ADMIN_PASS'))
        cy.contains('Log In').click()

        cy.contains('Dashboard').should('exist')

        // cy.activatePluginGui();
        cy.visit('/wp-admin/plugins.php')

        cy.get('#activate-plugin1').click()
        cy.contains('Thank you for installing plugin1 plugin!').should(
          'exist',
        )

        // cy.visitPluginSettings();
        cy.visit('/wp-admin/admin.php?page=plugin1')
        cy.contains('Create your account')
        .should('have.attr', 'href')
        .and(
          'include',
          'https://www.plugin1-staging.com/signup-wordpress',
        )
      })
    })
  })

  describe('w/ standard domain', () => {
    beforeEach(() => {
      // In this case it does the exact same as above with the exception that it sets a pre-defined domain
      // so it is recognized by the plugin. This domain does match the one given in cypress.config.js -> e2e.baseUrl
      cy.resetTestingEnv()
    })

    it('should display MS admin link when autoconfig succeeds', () => {
      // cy.adminAuth();
      cy.visit('/wp-admin')

      // eslint-disable-next-line cypress/no-unnecessary-waiting
      cy.wait(500) // The login page clears/resets the input from js whan loaded, so need to wait

      cy.contains('Username').type(Cypress.env('CYPRESS_WP_ADMIN_USER'))
      cy.contains('Password').type(Cypress.env('CYPRESS_WP_ADMIN_PASS'))
      cy.contains('Log In').click()

      cy.contains('Dashboard').should('exist')

      // cy.activatePluginGui();
      cy.visit('/wp-admin/plugins.php')

      cy.get('#activate-plugin1').click()
      cy.contains('Thank you for installing plugin1\'s plugin!').should(
        'exist',
      )

      // cy.visitPluginSettings();
      cy.visit('/wp-admin/admin.php?page=plugin1')
      cy.contains('Go to your plugin1 backend').should(
        'have.attr',
        'href',
        'https://admin.plugin1-staging.com/sites/145',
      )
    })
  })
})

Cypress Version (that I tested with and is broken)

10.3, 10.3.1, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.11, 12.2, 12.5.1, 12.17.0, 13.2.0, 13.6.2, 13.7.2

Other

I suspect that changing Cypress.config('baseUrl'); might be a contributing factor in this issue

Not sure how to create a demo test setup of this complexity (multiple domains, etc.) so please let me know what other information or debug output to provide in order to track down the issue

[AtofStryker]

Hi @denke8. Thank you for opening an issue. Are you able to provide a small reproduction repository that we could run to verify behavior? My guess is the random base url is having an effect here. We typically don't recommend setting the cy.origin to your base url. Since top is already set to that url, it shouldn't be necessary. But you can have a different base url per test. Have you tried getting this to work without cy.origin?

[denke8]

Hi @AtofStryker, Thanks for checking on this issue!

Are you able to provide a small reproduction repository that we could run to verify behavior?

Well, since I'm running against a WP installation, I'd need docker to be able to pull up:

• a db container (I.e., image: mariadb:latest)
• a wp container ( image: wordpress )
• a wp-cli container ( image: wordpress:cli )
  to set up the exact testing environment
  If this is OK? Can I use these in the reproduction repository?

My guess is the random base url is having an effect here. We typically don't recommend setting the cy.origin to your base url. Since top is already set to that url, it shouldn't be necessary. But you can have a different base url per test.

Thanks for the tip!

Have you tried getting this to work without cy.origin?

Based on your info above I have altered the test that it no longer sets Cypress.config('baseUrl', baseUrl);. Unfortunately the result is the exact same

[AtofStryker]

That is completely fine! As long as we have steps to set up the docker orchestration (or a docker-compose file) to get everything running and reproduce the issue.

[denke8]

Hi @AtofStryker, thanks for the patience!

I created the reptoduction repo based on the official reproduction repo:
https://github.com/denke8/experimental-origin-cookie
STR, A, E are in the readme of the repo

Note, I managed to score a single successful run in Electron, but it too fails 90% of the time. Chrome seems to 100% reproduce the issue, and FireFox (I only dud a couple of runs) also seems to fail

Again, thanks for checking on the issue!

[AtofStryker]

Hey @denke8. Thank you for creating a reproduction! I am trying to give it a run, but am getting the following issues:
.

I am on MacOS. Are you running on native linux?

[denke8]

Hey @AtofStryker,

Yes, I am on ubuntu 22.04, not sure what other steps are required for MacOS (like verify, etc.)

Please confirm that you did run npm run prepare:containers successfully beforehand

Also, you can run that script manually too from the shell like scripts/reset_testing_env.sh --base foobar.loc for more debug information

I have also just seen that Cypress 10.5.0 got released, so I'll check that out tomorrow and report back

[denke8]

I tried it with the latest Cypress version, 10.5.0, (Chrome 104) but unfortunately it did not fix the issue. I updated the repo to reflect this.

[AtofStryker]

Hey @denke8. I did run the npm run prepare:containers before launching cypress. I can try running running on my Linux Mint partition for a quick reprod, which I think is on mint version 21 (which should be Ubuntu 22).

[denke8]

@AtofStryker Sounds great, let me know if I can help in any other way .... video, debug output, etc.

[denke8]

@AtofStryker @rachelruderman
Could you please clarify if any additional information is needed from me in this issue?

Asking because the issue "timeline" states that the "Needs Reproduction" label was removed 5 days ago, but it still shows the flag as active for me. Not sure what's happening.

[rachelruderman]

Hi @denke8 , thank you for checking in and above all, thank you for your patience!

let me know if I can help in any other way .... video, debug output, etc.

These are always super helpful. You can find instructions here on how to print your debug logs

the "Needs Reproduction" label was removed 5 days ago

This was an oopsie, I just added it back on. I'm working on reproducing your issue, I hope to get back to you tomorrow 🙏

[denke8]

Hi @rachelruderman!
No problem at all, thanks for checking on this issue!

It seemed like a better approach not to mix in the regular output, so here is the debug log printed to stderr:
stderr.txt

And the stdout output was the following

Please note that there are a total of 3 test (attempts) in the log, in the following order:

• There is attempt 1 and 2 of the failing test (for some reason attempt 2 always seems to fail later than attempt 1)
• There is also the exact same test, but without the cy.origin, which does pass. The only difference is the domain set for the WP in the docker image, and the cy.origin stuff.

Hope this helps

export DEBUG=cypress:* && npm run cypress:run 2>./stderr.txt

> [email protected] cypress:run
> cypress run --browser chrome


====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        10.6.0                                                                         │
  │ Browser:        Chrome 104 (headless)                                                          │
  │ Node Version:   v16.17.0 (/usr/bin/node)                                                       │
  │ Specs:          1 found (manualActivation.cy.js)                                               │
  │ Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}                                            │
  │ Experiments:    experimentalSessionAndOrigin=true                                              │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  manualActivation.cy.js                                                          (1 of 1)


  Manual plugin activation
    w/ random domain
      (Attempt 1 of 2) should display signup CTA when autoconfig fails
      1) should display signup CTA when autoconfig fails
    w/ standard domain
      ✓ should display MS admin link when autoconfig succeeds (11780ms)


  1 passing (2m)
  1 failing

  1) Manual plugin activation
       w/ random domain
         should display signup CTA when autoconfig fails:
     AssertionError: Timed out retrying after 40000ms: Expected to find content: 'Plugins' but never did.
      at Context.eval (http://wp-plugin-e2e.loc/__cypress/tests?p=cypress/e2e/manualActivation.cy.js:111:10)




  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        2                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  2                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 minute, 52 seconds                                                             │
  │ Spec Ran:     manualActivation.cy.js                                                           │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Screenshots)

  -  /home/denke/work/320ny/experimental-origin-cookie/cypress/screenshots/manualActi     (1280x720)
     vation.cy.js/Manual plugin activation -- w random domain -- should display signu               
     p CTA when autoconfig fails (failed).png                                                       
  -  /home/denke/work/320ny/experimental-origin-cookie/cypress/screenshots/manualActi     (1280x720)
     vation.cy.js/Manual plugin activation -- w random domain -- should display signu               
     p CTA when autoconfig fails (failed) (attempt 2).png                                           


  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /home/denke/work/320ny/experimental-origin-cookie/cypress/v   (10 seconds)
                          ideos/manualActivation.cy.js.mp4                                          

    Compression progress:  100%

====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  manualActivation.cy.js                   01:52        2        1        1        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 1 failed (100%)                     01:52        2        1        1        -        -  

[rachelruderman]

Hi @denke8! I used a Linux VM to run your repro and I was able to reproduce the behavior you're seeing 🎉 Thank you for putting together a simple repro for us with clear steps in the README, it helps loads ❤

[denke8]

Hi @rachelruderman!

I'm glad that it helps, and thank you for checking this issue out!

[denke8]

@AtofStryker Thanks, appreciate the help!

[denke8]

Hi @AtofStryker,

I have updated the reproduction repo to 12.2.

It still does not work, but now WP flat out states that cookies are not supported by the browser

[goska-malaga]

It looks like I run into similar issue on cypress 12.3 (previously I tried cy.origin on 9.6 and it worked)
I am getting communication error that shows when cookies are expired (not sure if that is any helpful info that website is build on vaadin framework https://vaadin.com/), then I tried this company admin portal and it is also on WordPress so I run to same result as shown on the screenshot.
I do however run to this issue on the login page and one site blocks the login button on the wordpress page I actually didn't use correct login details but I get the same error anyway so it looks like it is easy to replicate. I got in fact impression that it worked just once as I approach to implement it and then as soon I started adding cy.session it broke and I keep trying to investigate it since last week.

And my steps are quite basic e.g.
cy.origin('https://portal.lobster.de/', () => {
cy.visit('https://portal.lobster.de/');
filling out fields and clicking the button

same steps works on other domains like aws

I find this interesting in replication step:
cy.wait(500) // The login page clears/resets the input from js whan loaded, so need to wait
as it takes indeed a noticable amount of time (other words it is visible slow to use cy.origin) and I wonder if there is anything related that when using experimentalModifyObstructiveThirdPartyCode: true it blocked the js so the login page didn't load at all (and it was cached quite well so I could not get rid of it for some time)

after all I am quite disappointed that feature that worked as experimental doesn't work when released and I cannot even run it on older version as the experimental flag no longer exists

[denke8]

Hi @AtofStryker,

I have updated the reproduction repo to 12.5.1.

Would you happen to have any updates for this?

[AtofStryker]

Hey @denke8. Sorry for the delay here. My linux machine is currently bricked due to some issues with lightdm, so I am trying to give this a go on my mac. I keep running into issues with the reset_testing_env.sh script with getopt for illegal options?

I tried switching from zsh to bash to verify in the terminal and still have the same issue

any ideas?

[denke8]

Hi @AtofStryker,

If I remember correctly I encountered the issue too, but the script did run correctly regardless. Please let me know if this is not the case

IIRC this error was only shown when the script was executed from under Cypress, but could not figure out the exact reason

[AtofStryker]

@denke8 I also seem to be running into it just through bash as well.

I tried setting failOnNonZeroExit: false for the exec command to see if I can sidestep the failure. We get further, but it looks like the WP server isn't set up and needs configuration?

[denke8]

@AtofStryker
This is very interesting. Yes, it seems it does not set up correctly in your machine.
The only thing I can think of, is a userId mismatch. I am running this from the latest LTS ubuntu (22.04), so I'm using 33:33 as user and group when executing wordpress-cli. I don't think that this should cause any problems, because we are talking about interaction within two docker containers, but still:

https://hub.docker.com/_/wordpress
please search for "Running as an arbitrary user"

Alternatively, I can create a VM image with a full linux install and docker if you want me to (and upload the disc file somewhere) ... not sure how else to proceed

[AtofStryker]

@denke8 the VM image might be the way to go since my personal linux machine is currently bricked and not sure what type of permission issues I am going to run into on my mac. I have a VMWare install on my machine and can run an ISO. Just shoot me a link if/when you're able to upload it!

[denke8]

@AtofStryker I have created an .ova file that you should be able to import. Do you have a way I can privately send you the link?

[AtofStryker]

can you send it to [email protected]? Let me know once you send it and I can ask support for it!

[denke8]

@AtofStryker

Sent, please look for an email with the title VM image - bug reproduction for #23165

[wizzyto12]

I have the exact same issue on a website. When entering the website, it creates a cookie PHPSESSID, which is not preserved while the login form is submitted and therefor the CSRF token fails.

[denke8]

Hi @AtofStryker
I have updated the reproduction repo to the latest 12.17 version, but the issue still persists.
Did you happened to have a chance to take a look?

[AtofStryker]

@denke8 I have the OVA file on hand but haven't had a chance to install the image and investigate. I'm hoping to have time on next rotation to dig in a bit more.

[denke8]

The issue unfortunately is still present, updated the reproduction to the latest version, 13.2.0

[kgartland-rstudio]

I'm hitting this too. Is there a workaround?

[denke8]

@kgartland-rstudio Not to my knowledge

@AtofStryker The reproduction repo has been updated to 13.6.2, the issue seems to persist.

[lennerd]

Having the same problem when testing a login feature with Keycloaks OAuth. Any updates?

[denke8]

@AtofStryker I have updated the reproduction repo to 13.7.2. Would you happen to have any updates on this issue?

[AtofStryker]

@denke8 unfortunately no updates on the issue. I appreciate you updating the reproduction though!

[AtofStryker]

gave this a go with the Cypress 14 Pre release Binary and the issue is still present unfortunately.

Note for self: Also, looks like the reason getopt has issues in mac with the script is it is BSD based and not GNU based. brew has a package gnu-getopt that can be referenced to get up and running on macOS.

[denke8]

@AtofStryker Thank you for taking an other look. Any ideas about the core issue?