Cypress experimentalMemoryManagement Experimental Feedback

[mschile]

Hey everyone 👋 , thanks for testing out the experimentalMemoryManagement feature to improve memory management within Chromium-based browsers!

We appreciate any and all feedback that you have to provide.

For any issues, please include the memory debug logs by performing one of the following steps:

• Set the CYPRESS_INTERNAL_MEMORY_SAVE_STATS environment variable to 1 or true and include the cypress\logs\memory\<spec name>.json file.
• Alternatively, you can run Cypress in debug mode setting DEBUG=cypress*memory.

If you have a larger bug to report, please file a dedicated issue.

[mi4i0]

Hi @mschile. I'm still facing chrome crashes with experimentalMemoryManagement. I collected logs for you. I hope it will be helpful
logs.zip

But I should note that it happens rarely.

numTestsKeptInMemory - default value
experimentalMemoryManagement - true

[mschile]

Thanks @mi4i0. A couple questions:

1. Are you experiencing the crash in cypress run mode, cypress open mode, or both?
2. Which spec is crashing?

[mi4i0]

1. run mode
2. will take a look later. But I guess last in json log file. It's not depend on specific spec. It's falling down in parallel runs

[musayyyib]

numTestsKeptInMemory - default value
experimentalMemoryManagement - true

Can you guide me in configuring these settings? I'm also experiencing an "experimentalMemoryManagement" error while running Cypress tests.

[mi4i0]

Second run logs with
numTestsKeptInMemory: 0
experimentalMemoryManagement: true
logs.zip

[bigcakes]

What kind of test run speed change are we expecting with this flag?

[mschile]

Internally, we did not observe any appreciable speed changes. With the flag on, garbage collection may be run if the memory exceeds the threshold between tests which typically took between 200-1000ms depending on how much garbage was being collected.

[bigcakes]

I did some very limited runs and saw a noticeable increase in testing time, but I will do some more extensive testing this week to see if it was possibly just some intermittent latency

[bigcakes]

I did a more extensive set of runs and it looks like on average that the times are within the margin of error with the experiment disabled and enabled

[thomergil]

Hi @mschile. Thanks for your work on this. It is a showstopper bug for me/us. Unfortunately, I don't see any change in behavior. It still crashes the same way. Setting CYPRESS_INTERNAL_MEMORY_SAVE_STATS doesn't create logs for me so I cannot send them. However, setting DEBUG=cypress*memory shows eventually shouldCollectGarbage: true, but it never goes back to false even on subsequent tests. If I'm reading the code correctly, that should trigger garbage collection on the next test. I likely misunderstand. Thank you.

[thomergil]

No crashes with numTestsKeptInMemory set to 0.

[thomergil]

@mschile -- just curious. Do you consider this a solution? This cripples my ability to look back at why tests failed.

[mschile]

@thomergil, no, I wouldn't consider this a long-term solution. I have some ideas to improve memory management in open mode. If you haven't already, feel free to play around with the numTestsKeptInMemory value. If you have longer tests, you may also see some benefit splitting the tests up into more targeted tests.

[thomergil]

Yep, I have. Thank you for working on this. I appreciate it, @mschile.

[thomergil]

Even with a value of 1, it (unsurprisingly) eventually crashes.

[jasminhusadzic]

Struggling with this since Cypress 11 appear. Finally in mine case it was solved with:

CYPRESS_NO_COMMAND_LOG=1

Increased performance
Lower execution times
No More crashes with cypress/included images

[thomergil]

@jasminhusadzic, maybe that is helpful in a headless scenario, but it makes debugging in open mode impossible. So this isn't a solution, unfortunately.

[hegeds]

I think this is an important pointer on the issue. While changing the behaviour, such that the command log is cleared in between test cases is a good step, but if one would have a long running test case, then one could reach the memory constraint. This is especially bad, as cypress promotes writing longer tests (for e2e)

[jasminhusadzic]

@thomergil yes, for headless scenario, used in CI run, witch is more important then open mode. Just saying in my case, this is helpful. And you can split config for open and run mode.

[MaximeCheramy]

Thanks, I had the issue since the upgrade to Cypress 13 and this seems to fix the issue (when run in the CI)...

[dominics]

• CRA monorepo, Webpack 4
• I'm still seeing crashes on v12.7.0, with numTestsKeptInMemory: 0 and experimentalMemoryManagement: true
• The site being tested is pretty heavy on resources (three.js, GPU usage, etc. - I know there's a handwavy "GPU stuff" mention in the list of potential memory crash causes), but I wasn't seeing crashes when the renderer memory percentage was high (memory usage would be a lot below the high-water mark, maybe ~30% or 40% during the crash)
  • Then again, maybe my app is allocating the other 70% really suddenly, wouldn't rule that out 😆
• Crash is only happening after re-running specs for a while, in the same cypress open session, and is visible with a single spec/on the first spec

I started to follow the instructions to isolate my spec (https://github.com/dominics/cypress-test-tiny/blob/master/cypress/e2e/spec.cy.js) BUT this separate repo seems more solid than my real one, on memory usage. And memory usage is consistently lower there (when testing the same page). That makes me suspect something like the Webpack watch (which is a major difference between the setups - and CRA does some pretty weird and wonderful stuff (like adding an entrypoint for the HMR)) - It does seem like the percentage used goes up on repeated runs of that example spec though - so maybe it's still useful for reproduction?

[hegeds]

Hey All,

I've created a repository where one could easily reproduce the issue, the TL:DR if too many command log is generated in test case the cypress runner still can crash. I deliberately created a small webserver against which the test can be executed.

https://github.com/hegeds/cypress-oom

[mschile]

Hi @hegeds 👋, thanks for moving some of your feedback to this discussion. We are aware of potential memory issues for long running single tests. This issue is being tracked on #4164. Unfortunately, I don't have a timeline of when it will be worked on but you can read more about our prioritization here.

[Chili82]

I installed V12.8.1 and turned experimentalMemoryManagement = true (numTestsKeptInMemory = 0), but I still get the Out of Memory issue. During the execution of the tests, I monitored the memory usage of the Cypress process and as long as the tests were executed properly there were no problems, but when the first test failed, it was executed again because retry = 1 was turned on. This is not a problem because the test should have failed, however the next test passed but the cypress process doubled the memory usage and all subsequent tests passed but the memory usage grew and after some time when the process reached 3GB Out Of Memory appeared.
After that, we turned off retry and quite a few more tests were performed, but again near the end of the feature, cypress crashed.

It seems to me that in the case when the test fails, there is a problem with memory leakage for some reason, especially if we have retry enabled, then the memory usage increases drastically and does not decrease during further execution.

[mschile]

Hi @Chili82 👋 and thanks for the feedback! Does the issue happen in run mode, open mode, or both? Are you able to provide the memory debug logs so I can visualize the memory usage for the browser renderer process?

For any issues, please include the memory debug logs by performing one of the following steps:

Set the CYPRESS_INTERNAL_MEMORY_SAVE_STATS environment variable to 1 or true and include the cypress\logs\memory<spec name>.json file.
Alternatively, you can run Cypress in debug mode setting DEBUG=cypress*memory.

[Chili82]

Hi, I've run tests in run mode. I will try to prepare a memory debug log. Thanks.

[tester-at-bmi]

(property) experimentalMemoryManagement: boolean
Argument of type '{ projectId: string; viewportWidth: number; viewportHeight: number; defaultCommandTimeout: number; requestTimeout: number; responseTimeout: number; fixturesFolder: string; downloadsFolder: string; ... 10 more ...; experimentalMemoryManagement: boolean; }' is not assignable to parameter of type 'ConfigOptions'.
Object literal may only specify known properties, and 'experimentalMemoryManagement' does not exist in type 'ConfigOptions'.ts(2345)

Using Cypress 12.8.1

[mschile]

Hi @tester-at-bmi 👋, the type for experimentalMemoryManagement is defined here. I have not been able to reproduce your issue, do you have a project I could look at?

[Palashctr]

I'm facing the same issue, and I've tried with the probable solutions, i.e. added these 2 commands in my config file - numTestsKeptInMemory: 0 // this is not a good practice because - this is blocking cypress ton take screenshots at every step - surely I won't use this.
experimentalMemoryManagement: true // We can keep this.

In some article I read that cypress 13.6.4 they tried to optimize this memory management, And according to that I've updated the version.
The problem is not solved for memory management and I've encountered with another problem - That is while running cypress test I can not switch my PC tabs - If I do so then cypress automatically block the test runs and the spec has a loading, that's it this session is gone or what I don't know - you need to run the specs again.

Again I downgraded the version, and the last problem solved. I can switch PC tabs.

Can anyone please help me regarding this?

[nhet]

Vereinbarungen Anlegen.json
numTestsKeptInMemory: 10
experimentalMemoryManagement: true
Using cypress: 13.6.6
NodeJS: 18.5.0
npm: 9.5.0

[jons-bakerhill]

We've been using Cypress for a few weeks now and it constantly crashes with "Aw, Snap!" "Error code: Out of Memory" after running a spec 2-3 times, obviously this is very frustrating. I enabled experimentalMemoryManagement and set the memory debug environment variable. The debug log is attached.

The test is only ~2 minutes long and a few dozen commands (although several times that in xhr requests) so it's rather frustrating that it crashes so quickly and so often.

ProductSmokeCreateExtension.spec.json

[Chili82]

I solved all the cypress problems, I switched to Playwright :). I'm joking, but it became difficult to search for the cause of the problem, I simply divided the scenarios into several smaller ones and started several threads at once, and that's enough for now.

[craigsumner]

I'm glad these memory logging options were added. I'm going to attach some results I've gathered from local experiments.
The main takeaway is that the command log is very expensive.
It's clear why CYPRESS_NO_COMMAND_LOG=1 solves these issues so well, in situations where we can afford to do without it.

The context of the test is a system with a complex, and very poorly optimized, form. The test takes about 4.5 minutes to fill out the form one time. I repeated ('burned') the test 10 times using cypress-grep, for a total time of 45 minutes.

Then I repeated the loop, but with command log disabled. That loop 'only' took 26 minutes.

Then I used jsontochart.com to plot the memory usage.
I deleted the infrequent elements (with the test names) from the json files, to make the data easer to chart. Sorry if that makes those logs less useful.

  │ Cypress:        13.13.3                                                                        │
  │ Browser:        Electron 118 (headless)                                                        │
  │ Node Version:   v16.17.0                  │

10-fold loop, with command log:

10-fold loop, no command log:

1 test, with command log:

1 test, no command log:

complex-form.with-command-log.burn-10.json

complex-form.no-command-log.burn-10.json

complex-form.with-command-log.burn-1.json

complex-form.no-command-log.burn-1.json

[jasminhusadzic]

Great charts. Command logs disabled are doing work so far.