High memory consumption on 1B model

[physimo]

I'm looking at about 22GB VRAM usage by using a 1B model (Llama 3.2 1b instruct Q3_M).
When using CPU and loading the model to RAM, it uses about 14GB RAM.

By the way, using different model like Llama 2 13B uses only about 17GB VRAM, which only makes this confusing.
Is there any setting or parameter that I'm missing here to make the 1B use low memory as what a 1B should be?

[StrangeBytesDev]

Try changing the context size. Llama 3.2 1B has a much larger context window than Llama 2 and context has a very large impact on memory usage.

[physimo]

Ah right, the 1B, 3B, and 11B has 128k context length, which explains why the VRAM usage is not that different across the parameter size. Thanks!

[giladgd]

node-llama-cpp attempts to use the largest context size that can be fitted optimally in the current hardware to allow you to process as much information as the model is capable of handling, but that also means that it can use a lot of VRAM.
If you don't need a large context size, you can limit it by using the contextSize option when calling createContext, e.g. .createContext({contextSize: {max: 4096}}).
You can also enable flash attention to have even smaller memory footprint without compromising on performance, but it may not work well with every model.

I'm working on incremental memory allocation, which will greatly help with this, but it may take some time since it's complicated to get it to work well with good performance.