Investigate memory leak

[vnugent]

The server crashes frequently in production. FYI we run 2 instances, 1.6 GB RAM each with heap size --max-old-space-size=950.
Todo:

• Use Chrome dev tool (or whatever available) to identify problematic areas in the code. No need to come up with a fix. We'll address it in separate PRs.

[CocoisBuggy]

I'm not totally sure just by looking at heaps initially, but there is one thing that is clear

Some ways are much more consistent at generating HUGE heap reports than others.

The first case is a load profile of about 40 VUs querying for single areas / climbs. This is enough load that not all requests are fulfilled and the server works hard, but the collector comes through at the end of the load test and clears away everything. I've tried to scale the tests such that they produce about the same traffic on the network, but clearly the second heavier test is just blowing up the whole room. So I'll quiet it down a bit.

part of the inherent memory cost here is that even though my gql request only asks for

query Areas {
      areas {
        area_name
      }
    }

the entire mongodb collection is loaded into memory every time I ask for it (ouchie)

// resolvers.ts
areas: async (
      _,
      { filter, sort }: { filter?: GQLFilter, sort?: Sort },
      { dataSources }: GQLContext
    ) => {
      const { areas } = dataSources
      const filtered = await areas.findAreasByFilter(filter)
      return filtered.collation({ locale: 'en' }).sort(sort).toArray()
    },

As can be seen in the above offending code.

This does not however help us understand why node fails to garbage collect this at the end of its run. If we capture the heap with its stack and a stripped back test we can find that pretty much all of our memory is tied up in the mongodb client queue (which we do not directly control, it is managed by mongoose.)

well, okay then. Let's upgrade to node22 and mongoose 8.14 (latest) and try again, maybe this is a bug that they already fixed

Nope. no such luck. So it's gotta be something we're doing (darn).

Summary of findings

• Leak seems to occur mostly in long queries that sped a lot of time in the toArray generator
• Have not observed leaks in queries that only fetch one document (even at enormous load volume)