How to load LLM with multiple GPUs

[yileitu]

I want to use Llama-2-13b-hf to do NER task.

[components.llm.task]
@llm_tasks = "spacy.NER.v3"

[components.llm.model]
@llm_models = "spacy.Llama2.v1"
name = "Llama-2-13b-hf"

But I only got several RTX3090 cards. How to configure to perform inference on multiple GPUs under the architecture of this spacy-llm library?

[rmitsch]

Hey @yileitu, spacy-llm wraps transformers for all open source models. AFAIK you'll need accelerate for multi-GPU inference, see here. This workflow is unfortunately not supported by spacy-llm at the moment.

[rmitsch]

See https://gist.github.com/mrcabbage972/d2cff4a0e856dd8e40e98d425f73802e (linked in https://medium.com/@mayvic/llm-multi-gpu-batch-inference-with-accelerate-edadbef3e239) for an example with accelerate.