Does memory usage of loadCollection() increase with the amount of non vector data?

[mtilsted]

Does memory usage of loadCollection() increase with the amount of non vector data?

That is: will storing other data such as the document text and other meta data increase memory usage?

I know that Milvus loads the entire collection before it can search it, but I can't find out if this is only the vector data, or if it also loads all other fields.

Edit:
I just looked at the Python documentation for the MilvusClient and that does have a skip_load_dynamic_field, so I guess the answer is "yes" anything is loaded unless configured not to. But this flag is not mentioned in the java sdk version of the MilvusClient.

I don't know why, but I always expected Milvus to only load the vector index into memory, and then fetch the extra fields from disk if needed for output. But that is apparently not possible so I guess I will store that data in my sql database, indexed by the Milvus row index.

So it might be worth mentioning in the documentation that all data is loaded, even if it is not not needed for searching.

[yhmo]

Yes, when loading a collection, Milvus loads all vector field indexes plus all scalar field data into memory by default. However, you can control which fields are loaded using the load_fields parameter to reduce memory usage: https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/load_collection.md

You can specify which fields to load like this:

client.load_collection(
    collection_name="XXX",
    load_fields=["my_id", "my_vector"] # Load only the specified fields
)

[yhmo]

The Java SDK supports partial load from v2.4.5: https://github.com/milvus-io/milvus-sdk-java/releases/tag/v2.4.5, which was not updated in the doc.

List<String> loadFields = new ArrayList<>();
loadFields.add("id");
loadFields.add("vector");
client.loadCollection(LoadCollectionReq.builder()
        .collectionName("XXX")
        .loadFields(loadFields)
        .build());