Sglang combined with ray distributed inference

[guleng]

Does SGLNG support the combination of ray distributed inference?

[zhyncs]

Currently, SGLang supports multi-GPU on a single machine and also supports multi-GPU on multiple machines. What additional benefits can integrating Ray bring? There are currently no plans for that.

[guleng]

@zhyncs Because I have found that it is very slow when testing multiple machines and multiple cards, even slower than a single machine with a single GPU. It takes about 54 seconds to write a 500 word script,

I start the multi machine and multi card command：

python3 -m sglang.launch_server --model-path /models/Qwen2-7B-Instruct --host 0.0.0.0 --port 30000 --nccl-init sgl-dev-0:30005 --nnodes 2 --node-rank 0 --tp 2 --mem-fraction-static 0.8

python3 -m sglang.launch_server --model-path /models/Qwen2-7B-Instruct --host 0.0.0.0 --port 30000 --nccl-init sgl-dev-0:30005 --nnodes 2 --node-rank 1 --tp 2 --mem-fraction-static 0.8

Under such startup conditions, the speed is particularly slow

[zhyncs]

In the example you provided, the size of the weights for Qwen 2 7B is such that a single machine with a single GPU is sufficient, there is no need for multiple machines or GPUs. I understand that using multiple machines or GPUs is typically done in scenarios where a single machine with multiple GPUs, such as Llama 3.1 405B, cannot accommodate the workload.

[zhyncs]

You should raise another issue and provide minimum reproduction demo.

[guleng]

But even if a single GPU card can start a GPU card, my multi machine and multi card distributed deployment cannot affect its efficiency, right?

[zhyncs]

Nope. You should consider the communication overhead.

[guleng]

The reason why we consider using kuberay when sglang supports multiple machines and cards is that ray supports model fine-tuning, distributed task scheduling, and other functions. We greatly appreciate sglang's inference speed, and it would be even better if it supported kuberay