Llama-server functionality with rpc support?

[justinjja]

Trying to do multi node inference,
Have it working with llama-cli, but is there a way to do it with llama-server / an openai endpoint?

[ggerganov]

I believe it should just work - what command are you trying?

[justinjja]

Not sure what I was doing before, but I went to recreate this and it just worked.
Thanks!

[segmond]

I read that both systems need to have the model weights, so transfer over network can be a huge bottleneck. Would it be possible for both models to have a copy of their own weight, do a checksum to make sure it's the same, then the RPC server will just be communicating which parts of the weight each client needs to load? This should allow for faster loading across even a basic network instead of possibly an hour wait time for a very large model.

[lexasub]

@segmond full model downloading - it's bad) disks in remote hosts is not infinity, but split model on files & load in model - it's global pull request with type project)