Different Models For Multiple Assistants

[ChadwickTheCrab]

Hey! I've been having fun playing with this. I made two assistants in HASS each with a different model filename. When I switch between the assistants, it doesn't appear to switch the model. Whatever model is set in Text Generation WebUI is the model regardless of which assistant is currently selected. Did I do something wrong?

[markmghali]

That is not how this integration is designed to work at this time. IT only verifies what model is loaded in textgen webui . It does not switch them for you at this time. you would manually have to load it in textgen webui

[Anto79-ops]

@chadblose try LocalAI for your backend, and then when you go to add integration (or add service if you already have a Llama config setup) in HA, make one for each different model. Then when you set up a the voice assistant, chose whichever of these models you want and they will be 2 (or more) seperate models for Home-LLM Lllama

[colaborat0r]

Really? Why do we have to specify the model in the integration's settings then? (Either way is fine with me.)

[Anto79-ops]

well, this gives you choice, for example, if you can somehome create a script that can choose between the two, like for general chat send request to Mixtral 7x8b model, but to control your house, send it to home-llm-3b mode...acon96 do you think should could be something to play with? this way your model can specifically do HA function calls (and do it well) and another model can handle everything else...this could delete the purpose for trying to to train a bigger model to do everything, which would require more hardware/power....?

[acon96]

To fully close this out:

This component already fully supports multiple models at once and is just a limitation of text-generation-webui, which only supports having one model loaded at once. If you want to use multiple models at once then host multiple separate instances of text-generation-webui or a use different backend that supports multiple models loaded at once.

[colaborat0r]

Understood.