ChunkLoadError > Loading chunk cypress-support-file failed #28644
Comments
|
We made an update from webpack 4 to webpack 5 in 12.17.4, I wonder if it has anything to do with that upgrade. https://docs.cypress.io/guides/references/changelog#12-17-4 |
jennifer-shehane
added
the
stage: needs information
|
We are getting the same error also intermittently. I don't think we had it as far back as 12.17.4 but I can't say that for certain. It only seems to happen toward the end of our Component Test runs. We have 1500+ component tests that we are running. |
|
Same for us too |
|
We have been running into this sporadically when Component testing after migrating from CY 12.x to 13.6.2. Similarly to cases above, it happens towards the end of running a batch of suites (files) together.
Resorted to running cypress with There is a lot in the logs that makes it hard to share but please do let us know if we should be looking for or can share anything useful to help unraveling this. |
|
Does anyone know of a way to suppress this error until we can get a resolution? |
|
We have a very similar issue with another file: It makes our test suite pass sometimes but fail randomly and frequently, on a component or another, randomly. |
|
I'm leaving behind some notes after digging into this further. We are experiencing this error on ~9% of our component test builds. (I can't remember offhand how many tests that entails.) Indeed error messages above are consistent with what we see, here's an excerpt (with a slight edit on chunk name tbh): It includes the localhost path as well on the next line (which I left out here). Noteworthy...going up a few lines in the build, we are seeing something like this (I changed more file names out of caution): It appears to me that the Ultimately I'm convinced the next test needs to wait until compilation completes and the devserver has updated the cache. I attempted to set devserver settings around hot reload, etc, but those didn't work -- likely because the test is still trying to serve the page and is not waiting for the devserver to signal it's up to date. When it comes to waiting for compilation to complete, we do have logic around this in There had been a previous PR that leveraged the webpack devserver's index HTML file, which I believe would yield the result I'm thinking of (change here). That PR was intended to fix a situation specific to MacOS version, but I'm optimistic it could resolve the ChunkLoadError. The PR was merged but the code in the file today is older than the PR, so I don't know if it was rolled back at a later time. |
|
Just commenting to say I am also experiencing this issue. Have traced it down to being introduced by 12.17.4, any version below this seems to be completely fine on our CI runs. |
|
We're spiking into this to see if we can find what's causing this. |
jennifer-shehane
added
stage: investigating
|
Same here. Since the v12 upgrade (I don't recall what the version was, though), we have gotten this on GitHub Actions CI sometimes. My initial thought was that it was caused by the first spec in our test suite. However, changing the order of suites didn't help. It's just that the first spec fails. My workaround for this was to amend the run script: Cypress 13.6.4 BTW, this seems to be related or even the same: #16421 |
|
Hey all. Sorry for the delay on an update here. I've been investigating the issue heads down for about a week now. I'm able to reproduce the error on my end but only at the start of the run and sparingly in the middle of the run. I think there are two issues possibly going on here At the start of the runAt the end of In other words, we can get ourselves into a race condition where
In the middle of a runAdding to what @benpatterson mentioned and a bit more complicated to reproduce, but does happen, is Test starts. Something triggers
|
|
@AtofStryker, great investigation and summary. I can confirm that the error happens only at the beginning of the run for me; I haven't noticed it in the middle. I also haven't noticed any webpack re-compilation in the middle of the run. The compilation times on CI are around 19 seconds for the first run (which is allowed to fail) and 14 seconds for the second run. |
|
@bene-starzengruber @cubanx @werge2121 @amir1218 @rbecheras @TomaszG what version of I think the recompiling in the middle is related to either a bug or a misconfiguration within @TomaszG I would be curious if using the diagnostic binary while setting |
|
We should just be using the default devserver that comes with cypress, so 4. For us its happening in the middle of the run usually. It takes around 5-6 seconds for the webpack to compile usually. |
|
My team is using the default dev server that comes with cypress: The errors seem to always happen at the start of a test file, but pop up in random places in our test suite. For example, here is a recent failure in the 73th test file out of a suite of 90 where the other 89 runs all passed (edited to remove component/file names): The frequency with which these errors occur has been increasing as we add more test files. If there's any debug/diagnostic config we can set to get more info happy to try as this issue is happening frequently in our CI |
|
It would be great if cypress team would develop a pre-bundle option for runs in the CI. We had similar issues with MSW in our component tests. Browser stops trying when a service worker script takes more than ~60 seconds to load. We had to slightly modify the default dev server to "prebuild" service worker file so that when it's actually needed, it's already built and is ready to serve by the dev server: So if there were an option to pre-bundle everything, flaky tests due to middle of the run builds taking too long would be long gone. In user land, this can be achieved by manually pre-bundling and then by creating a custom dev-server that serves pre-bundled files. But it's not very elegant, Cypress team can do better by introducing a new config for pre-bundling. |
|
Hey all. Just a few updates. Mid recompilation with This issue has been extremely challenging to reproduce, but we have a few theories. My guess is that there are chunk loading issues associated with the main bundle mainly due to us bundling every component test, regardless of what is used under test. The only exception to this is if the
The larger In most cases, even if the build takes minutes, cypress waits for the dev server to return a response before executing tests. @anilanar In other words, we are pre-bundling everything needed under test already but the assets are not loaded into the browser all at once, which might explain the service worker issue you are running into and I'm glad you were able to find a solution! Something is causing the chunk to fail to load meaning it either doesn't exist, or it timed out loading. Since the assets are held in memory, it is difficult to figure out which has occurred. Either way, I am not convinced flaky tests are due to builds taking too long. What we are trying internally with one of our projects where this error happens frequently, is to leverage the |
|
@AtofStryker We are using whatever I concur that it seems large tests suites seem to cause this issue. To both speed up our test runs and reduce the frequency of this error, I have some PowerShell scripts that use the This reduced the number of test per run from ~1500 to ~400 per run, and the error seems to have abated. If this helps at all, even running interactively, if we load all our Component tests and then try to run them, there is a significant delay between update/reloads or if you switch test files. I use another PowerShell script to modify the We also tried switching the Component Server in the past to I am actually about to leave the current position I'm in, but if there's any diagnostics or info that would help you, I'll be here until March 1st. Thanks for your work on this! |
|
@AtofStryker Webpack supports lazy loading with async chunks (code splitting) therefore it is definitely possible to make a test suite start downloading and bundling stuff in the middle of the run. Webpack dev server doesn't initiate building an async chunk unless that chunk is requested by the client with an async import. This issue can be solved with pre-bundling. It can probably be solved without pre-bundling too, if there is a way for webpack dev server to report any async chunks it finds during a build. Then those async chunks can be requested manually (like I did with the service worker) to manually trigger a build for them before running a test. Keep in mind that this needs to be done recursively, until no async chunks are left. Another idea is to disable code splitting, e.g. https://webpack.js.org/plugins/limit-chunk-count-plugin/ I'm not really sure if this has anything to do with your A simple test that downloads an async chunk in the middle of the run: export const Example = () => {
const [done, setDone] = useState(false);
return (
<button
onClick={async () => {
await import("./path-to-something-very-slow-to-bundle");
setDone(true);
}}
>
{done ? "DONE" : "Click me to download async chunk"}
</button>
);
};it('will download async chunk', () => {
cy.mount(<Example />);
cy.get('button').click();
cy.contains('DONE').should('be.visible');
});Edit: I still think it's better to pre-bundle and run cypress without a dev server and without a file watcher and without no recompiles for CI use cases; instead of trying to fight file watcher/unnecessary rebuilds that can be caused by myriads of webpack plugins (and potentially unintentional bugs in them). I think using dev-server on CI is picking a wrong fight. Consider this: no build tool or no test tool runs in watch mode on CI. So cypress is opening a can of worms by running in watch mode. |
|
@AtofStryker We are on webpack-dev-server 4.15.1 |
|
Any updates? |
|
Try this workaround, as I previously suggested: // cypress.config.js
const { defineConfig } = require('cypress');
const { devServer } = require('@cypress/webpack-dev-server');
module.exports = defineConfig({
...
devServer: async ({ specs, cypressConfig, devServerEvents }) => {
const result = await devServer({
specs,
cypressConfig,
devServerEvents,
webpackConfig,
});
// prebuild files that can cause test failures when they take too long
// during runs
await Promise.all([
fetch('http://localhost:8080/__cypress/src/cypress-support-file.js'),
// prebuild any other file, such as service workers here
]);
return result;
},
});If other async chunks are causing middle-of-the-run builds, then try using https://webpack.js.org/plugins/limit-chunk-count-plugin/ and limit your chunk count to 1. Or try turning off chunk splitting, see https://webpack.js.org/plugins/split-chunks-plugin/#optimizationsplitchunks . Again, I doubt this is a misconfiguration in webpack dev server. Async chunks are normal things that optimize initial bundle size by lazy loading things later. They are usually caused by async imports, but if you don't have an async import on your side or if you don't know what that even means, then it must be used by one of the libraries you use. This is a problem with async chunks triggering builds and those builds not finishing before test times out. The solution is one of:
Edit: typo |
|
@AtofStryker I've tried what you've recommended, so I've set CYPRESS_INTERNAL_BROWSER_CONNECT_TIMEOUT=120000 and also used your diagnostic binary based on 13.6.5 and after a couple of runs, it seems that there are no failures. Here's the log output from the tests: I wonder if it's OK that |
|
@rgala are you referencing import { devServer } from '@cypress/webpack-dev-server'
export default {
component: {
devServer: (devServerConfig) => {
return devServer({
...devServerConfig,
framework: 'react',
bundler: 'webpack',
})
}
} |
|
Sorry @AtofStryker, I did not know I had to make such change. Anyway, after implementing it I have been unable to reproduce the issue anymore and this is good news :) |
|
We intend to make the just-in-time compile flag default within Cypress 14, with a |
|
@AtofStryker following up on the issue I reported related to @AtofStryker Can you please answer this ? Your previous response can be seen here. Thanks |
|
I am experiencing this issue in gitlabci on So in the end im not even using the experimental JIT flag. But I'm not super familiar with webpack so I'm a little unsure about what this is actually doing or if there could be negative ramifications. |
|
We're on 13.14.0 and since the upgrade we have not had any more occurrences of chuck load error. Sounds like we should hold off on 13.15? |
|
|
With larger codebases, the |
@TomaszG Execution time is going to vary on several factors, but in some cases it is completely reasonable to expect longer execution times. Are you leveraging any type of smart orchestration features, like parallelization to speed up the total run time? Most importantly, are the chunk load errors still present with |
|
@AtofStryker Will the patch solution you provided be implemented in future versions or I should stick to it as long as it helps with my Azure builds? |
@rgala I would stick to using the patch for now. We are currently discussing solutions as the sourcemaps do provide value in the Cypress Reporter for error code frames (where your test failed) as well as in test replay |
|
@AtofStryker Can you please answer this question incase you have missed ? |
Sorry @papatelst I must have missed it. The |
|
@AtofStryker Can you give me exact steps on how to answer your question ? On one of the We also saw this error in a cypress run with specPattern which returns 37 spec files. Not sure if this has to do with the number of spec files found during cypress run or not. |
I think what I am moreso wondering is how many tests are in that spec file that is problematic. Is it always the same spec file or does it happen with others as well? Are you able to create a separate issue as well against the Cypress repo? I believe what you are experiencing is going to be outside the realm of chunk load errors related to CT testing with webpack |
|
@AtofStryker It happens randomly with random spec files. It does not happen with same spec files. |
If it's happening with end to end test then I would open a separate issue |
Yes, we do run the tests in parallel and component tests run takes significantly longer, especially on CI. Given that webpack bundle doesn't change during the test execution, I don't see a reason of involving webpack bundle for each spec, as that's a major performance drawback, especially when testing complex enterprise apps. I recon that a simple wait (even static if there's no milestone on which you could rely) for webpack to finish or a one-time retry in case of ChunkLoadError would make the error gone for most of the users. |
|
What I found out is that Cypress ignores the I tried to configure retries in different ways, but it's not retrying: export default defineConfig({
retries: {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 2,
passesRequired: 1,
},
runMode: true,
openMode: false,
},
component: {
…Can't we make it to retry the component test on such an error? Not only on a flaky test. Perhaps, in the next retry attempt, the chunk is already there (compilation finishes). |
It's likely more difficult to do in this case because the chunk is likely failing to load due to resource constraints and likely retrying to load it will not work. Have you tried the |
|
@AtofStryker Enabling In larger projects, it causes major slowdowns in test execution on CI and increased memory requirements. We now have to run 6 GB RAM machines, instead of 4 GB, to stay stable, which is 1.5x as expensive on providers like CircleCI. This makes a pretty significant difference in our CI budget, given the sheer number of component tests we run. As much as we'd like to stay with Cypress, the ever-increasing memory requirements and instability are really putting into question whether investing in component tests was a good idea to begin with, compared to something like Vittest. Have you considered migrating to a webpack alternative, while maintaining the ability to mock/stub imports? Could that make the JIT part easier on the memory and time? |
|
@shentao Just want to note that the justInTimeCompile configuration will be available to opt out of in Cypress 14 if needed. |
@TomaszG We initially tried this when exploring options for a solution here. The problem is the assumption that the ChunkLoadError can be reconciled and that repeatedly calling the entrypoint would eventually fix this. Depending on your webpack configuration, it could, but it doesn't always work unfortunately and is extremely error prone because usually the reason the chunk is failing to load is the dev server eating too much memory and resources because of how large the bundle is. Often, the process completely stalls. At least this is what I saw when using one of our really large projects internally and resource constrained machines. Another solution could be to pre-bundle similar to what @anilanar did in #28644 (comment). To their point, we have seen that source maps, depending on the Component Testing test harness, can be quite large and can additionally lead to memory problems (see #28644 (comment)). We want to possibly add the option to disable source maps in CI in the future along with adding some tools to better help diagnose user issues (see #30461). We are hoping with |
|
@jennifer-shehane I know, I've been following this thread for a long time now. The problem is that, with |
@AtofStryker That's something that CI providers often provide out of the box (load balancing tests based on timings). It might help with the overall time, but it does nothing to help with the budget side of things. Throwing more machines at the problem should not be the only solution to a performance problem like this. Even if it is the simplest way to go around it, compute is ultimately not free. Unless of course, Cypress starts offering alternative hosted CI runners where clients can pay on a per test basis rather than per compute minutes. |
|
@shentao Just to be clear are you saying you were using 4GB machines before JIT was released and now you are up to 6 GB? I find it hard to believe that JIT would take more memory than the default implementation. That would suggest there is a memory leak someplace and I would be curious to investigate with some type of reproduction. To your point, compute is not free, but horizontal scaling gives you options depending on what you need. We can only work with the vertical limit we are given, which is webpack dev server's ceiling to host very large bundles. The lower (or lowest) we can make that vertical ceiling, the better scale options we can allow users to partake in. It gives consumers a choice to possibly use one large machine, or several smaller ones in parallel. Or having the option on with no chunk errors on a single machine, but the tests take a long time to complete . Compute may be cheaper, but the tests could take longer to run. It's a sliding scale on how the consumer wishes to orchestrate their CI based on what factors matter most to them in the tradeoff: compute cost vs time for tests to complete. To be clear, you do not have to use Webpack with Component Testing if you do not wish to. We fully support Vite to bundle your components and run them in the browser. Our vite dev server uses file system references to hydrate components in the browser, which makes it much faster to run generally. There are many reasons you would want to use some type of browser based component testing which we try to articulate in this blog post, but just like anything there are trade-offs, which are most obvious between something like a node context runner like vitest with jsdom, and browser based testing like with Cypress. |
|
I get that webpack is theoretically not a hard requirement (mentioned that in a comment earlier) except that it is because without it the mocking/stubing functionality is not working with Vite due to the nature of esm. At least that was a problem a few months ago when I last tried to migrate away from using the webpack dev server. That's actually the only reason why we still have webpack in our project - to support Cypress component tests. The app itself is built with Vite. If anything changed in that regard, I'm happy to try migrating again. I would not rule out a memory leak somewhere – the project consists of almost 2000 Vue components and a similar number of ts and js files. But investigating that is not an easy task as you might imagine. Don't get me wrong, I've been a proponent of Cypress pretty much since its inception. But the amount of frustration it has been generating for our team over the last years is beginning to be hard to ignore. Of course, a lot of that might come from the pains of a rapidly growing codebase. So please do take my rant with a grain of salt. 😅 edit: we are also using the code coverage plugin for component tests which is contributing to the memory consumption, though I haven't measured exactly by how much. |
and others
Current behavior
After updating from
[email protected]to[email protected], we are seeing this error occasionally on our CI runs.Unfortunately it is not reproducable locally, so I can not give you a repository to check.
I know that this might make it hard to investigate but maybe somebody already fell over the same issue or others are experiencing it as well.
The test fails around 30% of the time.
Desired behavior
No response
Test code to reproduce
no reproduction repository
Cypress Version
13.6.2
Node version
18.19.0
Operating System
Ubuntu 22.04.3
Debug Logs
No response
Other
No response
The text was updated successfully, but these errors were encountered: