Look like snowpack loop and leak

[Akiyamka]

 npx snowpack --reload --verbose build

[snowpack] ! clearing cache...
[snowpack] ! installing dependencies…

<--- Last few GCs --->

[654077:0x5fcda20]   140328 ms: Scavenge 2033.2 (2057.3) -> 2027.2 (2057.6) MB, 18.5 / 0.0 ms  (average mu = 0.276, current mu = 0.219) task 
[654077:0x5fcda20]   140349 ms: Scavenge 2033.6 (2057.6) -> 2027.6 (2058.3) MB, 2.9 / 0.0 ms  (average mu = 0.276, current mu = 0.219) task 
[654077:0x5fcda20]   140368 ms: Scavenge 2034.0 (2058.3) -> 2028.0 (2058.8) MB, 2.8 / 0.0 ms  (average mu = 0.276, current mu = 0.219) task 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa3ac30 node::Abort() [node]
 2: 0x98a45d node::FatalError(char const*, char const*) [node]
 3: 0xbae25e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xbae5d7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd56125  [node]
 6: 0xd56acb v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd6481c v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd65684 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xdc025f v8::internal::ScavengeJob::Task::RunInternal() [node]
10: 0xca125b non-virtual thunk to v8::internal::CancelableTask::Run() [node]
11: 0xaa84b4 node::PerIsolatePlatformData::RunForegroundTask(std::unique_ptr<v8::Task, std::default_delete<v8::Task> >) [node]
12: 0xaaa319 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [node]
13: 0x136f626  [node]
14: 0x1381ae0  [node]
15: 0x136ff58 uv_run [node]
16: 0xa7ab94 node::NodeMainInstance::Run() [node]
17: 0xa0a861 node::Start(int, char**) [node]
18: 0x7fb0dd4fb042 __libc_start_main [/lib64/libc.so.6]
19: 0x9a526c  [node]
[1]    654077 abort (core dumped)  npx snowpack --reload --verbose build

Currently I try to investigate what exactly loop snowpack.
Verbose mode not enough verbose )

[FredKSchott]

Hmm, that's unexpected!

Do you define any "mount" config in your Snowpack config file? By default, we assume the entire working directory is a static app, and would be scanning your entire directory for JS source code. If your directory contains many non-frontend files, that could be causing the issue.

[Akiyamka]

I have not used any files other than the frontend related, just SPA. However, you're right about not using the mount setting!

[kevinbarabash]

I'm running into this too. I think the issue is with rollup trying to do too much at once. Instead of passing to all modules at once to rollup to build, maybe we could build one at a time with separate calls to rollup.

[FredKSchott]

When we've seen this in the past, it's almost always because we are scanning too many files and not because of the Rollup install itself. Since we're dealing with frontend code, it's rare to have more bytes of frontend code than Node.js has memory.

Like @Akiyamka, most likely this is caused by having lots of non-frontend files in your current working directory, and not using a "mount" config to specify which files are actually needed.

[FredKSchott]

oh wow, that's surprising and I stand corrected!

You can try running Snowpack like this: NODE_OPTIONS="--max-old-space-size=4096" snowpack build to see what the output is when you bump the memory up. I'd love to see the output of this working if you could share it.

Otherwise, we may need a heapdump to check out what exactly is eating up all your memory. You would need to add some profiling code to your local copy of Snowpack, see: https://medium.com/better-programming/make-a-dump-of-the-v8-heap-and-inspect-for-your-node-app-b69f7b68c162

[lewisl9029]

Found this thread after running into the same out-of-memory issue here, trying to migrate a very large project over to snowpack, and can confirm moving to individual module builds resolved it (though other issues popped up afterwards, still looking into those, might need to start a thread for help later). For a sense of scale, the project has over 2gbs of files in node_modules, and I was still getting oom errors after bumping heap size to 8gb.

This makes me wonder if snowpack even should be trying to build every single dependency in the first place? There are many dependencies in our package.json that are used by a backend service in the same project (this is definitely not ideal, but it is what I'm dealing with here).

Webpack is able to trace the dependency tree from the entry point for each app so the backend app never touches frontend dependencies and vice versa for the frontend app. Could snowpack not do the same so it only installs dependencies it encounters on the path of resolution from the scripts in the index.html entry point?

[lewisl9029]

The downside though is that rollup won't automatically create common bundles from shared dependencies.

Looks like this is coming back to bite me now. After powering through all the other issues documented in #1387, I'm now running into "Invalid hook call" errors in react, presumably because of the multiple copies of react from transitive dependencies as all my dependencies were built independently.

I'll revert back to the single rollup call and see if I can capture some debugging info somehow.

[lewisl9029]

Reporting back, things ended up working after I bumped the heap size to 16gb, which feels excessive, but was enough to unblock the migration. I'll try to follow the links above to produce some memory dumps and see if I can find any patterns sometime next week.

[FredKSchott]

That would be great, thank you! I've tried looking into each Rollup plugin for memory leaks but nothing jumps out, so it seems like something may be happening with Rollup internally (possibly related to source maps?). Would love any info you can find via memory dump 👍

[yesmeck]

I'm running into this too, here is the huge heap snapshot file https://www.dropbox.com/s/jsb57csef7vw3vs/Heap-20201111T193506.heapsnapshot?dl=0

[meh]

This is also happening to me now, after changing some dependencies it started happening, anything I can do to help here?

[winkler1]

@FredKSchott Is there anything you need on this issue? Looks like there's a 2.6GB snapshot posted - #857 (comment)
I'll be rolling out Snowpack to co-workers on 16GB machines, will see how it goes on their machines.

Feels like something is improved on (both 3.0rc1 and and 2.18.4) -- used to have to go to 16384, now export NODE_OPTIONS='--max-old-space-size=1500' && ./node_modules/.bin/snowpack dev --reload works.