What is the usage of -spill_path?

[qiranq99]

Hi!

From the information provided by python3 -m vineyard --help, the option -spill_path <path> seems to allow spilling data to the disk storage if there is no sufficient memory. However, the program gives not-enough-memory once the allocated shared memory has reached maximum capacity, instead of trying to spill the data with a server started via python3 -m vineyard -spill_path /.../...

Therefore, I'm wondering the actual usage of -spill_path, or any substitutions for spilling the memory.

Cheers :)

[dashanji]

Hi, @qiranq99. Thanks for the report.

From the information provided by python3 -m vineyard --help, the option -spill_path seems to allow spilling data to the disk storage if there is no sufficient memory.

Yes, you are right. The spill mechanism is for vineyard server. In theory, the total amount of objects that vineyard server can store is at least shared memory size + spill path size (objects deserialized to disk are smaller than the memory in general).

However, the program gives not-enough-memory once the allocated shared memory has reached maximum capacity, instead of trying to spill the data with a server started via python3 -m vineyard -spill_path /.../...

When using the python3 -m vineyard -spill_path /.../..., the vineyard will use the default share memory size 256Mi. Thus, one client can only use no more than 256Mi shared memory, If it exceeds, the error will be triggered.

You can create another client (notice, avoid client cache here) and put objects in the vineyard, then the previous objects will be deserialized under the spill path. Also, you may get some inspiration from the spill test. https://github.com/v6d-io/v6d/blob/main/test/spill_test.cc

BTW, could you please share your use case here? Thus, we can make the spill mechansim better.

[qiranq99]

@dashanji Sorry for not making it clear. The scenario is like:

1. start a server via python3 -m vineyard -size 100G -spill_path <path>;
2. A bunch of clients continuously putting large data objects, e.g., 10 G, into shm;
3. Another client would invoke clear() periodically.

With the periodical clear() invokes, the shm would be released for storing incoming data. However, occasionally the shm usage exceeds 100 G, while it is supposed to be allowed in the use case that we might want to put 110G data and clear them later. In such a use case, the requirement would be spilling the additional 10G data to the storage path to make them retrievable. I'm not sure if -spill_path is meant for it.

[dashanji]

@dashanji Sorry for not making it clear. The scenario is like:

1. start a server via python3 -m vineyard -size 100G -spill_path <path>;
2. A bunch of clients continuously putting large data objects, e.g., 10 G, into shm;
3. Another client would invoke clear() periodically.

With the periodical clear() invokes, the shm would be released for storing incoming data. However, occasionally the shm usage exceeds 100 G, while it is supposed to be allowed in the use case that we might want to put 110G data and clear them later. In such a use case, the requirement would be spilling the additional 10G data to the storage path to make them retrievable. I'm not sure if -spill_path is meant for it.

Yes, you are right. The spill will spill the additional data to the storage and reload it to memory when necessary. Also, the spill has two watermarks spill_upper_rate and spill_lower_rate as follows. Supposed we use the default value and the shm size is 100G, when the shm usage exceeds 80G (100 G * 0.8), the vineyard will spill the exceeded data to the storage path until the shm usage is 30G (100G * 0.3).

python3 -m vineyard --help
vineyardd: Usage: vineyardd [options]
...
    -spill_lower_rate (low watermark of triggering memory spilling)
      type: double default: 0.29999999999999999
    -spill_path (path to spill temporary files, if not set, spilling will be
      disabled) type: string default: ""
    -spill_upper_rate (high watermark of triggering memory spilling)
      type: double default: 0.80000000000000004
...

[qiranq99]

Hi @dashanji

Please try the following use case with python -m vineyard -size=50G -spill_path=<whatever>:

import vineyard
import numpy as np

data = np.random.rand(1000,1000,1000) # ~7.4G each data object
client = vineyard.connect()
for epoch in range(10):
     for batch in range(10):
          client.put(data)
     client.clear()

I would expect $74-50=24G$ data to be spilled to the disk, but in reality it gives me: Not enough memory.

Not sure whether it's a bug or not.

[dashanji]

@qiranq99 It's not a bug. As you set the size=50G for vineyard, that means the maximum usage memory of client is 50G as well. In your case, the memory usage of client is more than 50Gi.

To test the spill, you can try the following code.

import vineyard
import numpy as np

data = np.random.rand(1000,1000,1000) # ~7.4G each data object
for epoch in range(10):
     client = vineyard.connect("Use the default vineyard socket here")
     # one client can't put more than 50G data
     for batch in range(5):
          client.put(data)
     client.close()

Then you will find the serialized the objects in the spill path.

[qiranq99]

@dashanji Theoretically, if vineyard server has a setup of -size=75G -spill_path=/folder -spill_lower_rate 0.1 -spill_upper_rate 0.4, I would expect the follows when running 10 batches:

1. all data is cached and retrievable, which is true;
2. when monitering the system shm usage using watch -n 1 free -g, the peak utilized shm should be 75*0.4=30 G. However, the maximum usage of shm is observed to be 75 G;
3. disk usage on the spilled path should be observed, but in reality the folder is empty and the serialized data objects are invisible.

Basically, by utilizing memory spilling, we want vineyard to apply for at most 30G shm to the OS for saving resource.

[sighingnow]

Ideally, the provided example should work. I'm not sure if there are any regression bugs happen, and spill is not triggered here. We'll double-check.

The memory usage might exceed 75G * 0.4 shortly, as the spill is triggered at the next allocation. Anyway, with spill enabled, putting those ten tensors with --size=75G shouldn't fail, IMO.

[sighingnow]

There're indeed some lifetime issues requires further investigation.