Home Assistant 2024.6 Better LLM Support

[markmghali]

Hello,

Love your integration and all the work you have been doing! I saw in the beta release notes for HA 2024.6 they are now allowing LLMs to control Home Assistant. I am wondering how this integration can get a jumpstart on these new features?

See Beta Release Notes:

https://rc.home-assistant.io/blog/2024/05/29/release-20246/#dipping-our-toes-in-the-world-of-ai-using-llms

[Anto79-ops]

at the moment, its only Google Generative AI and OpenAI.

In a future release, it will be local LLMs

[acon96]

Yeah I have a branch I'm working on to get support added. I should push it up to github once it's a bit more polished.

There is a pretty significant amount of work to be done to refactor this integration to follow the new API patterns as well as refactor the dataset generation scripts so that I can train a new set of models that would work with the new APIs.

The issue is that the APIs being added take a completely different mindset/methodology for prompting the models that is very different from how this integration currently prompts models. The new APIs that HA is introducing seem to be following the OpenAI function calling standards that I have explicitly avoided because they are overly verbose (they use giant JSON objects using lots of tokens) which is only useful if you have a giant GPU to run your model or lots of time to wait around for an answer. This is explicitly BAD for a real-time voice assistant that is running locally on lower-spec hardware (i.e. CPU). Most of the features in this integration are focused around reducing the number of tokens that need to be processed to ensure the fastest response time possible.

I do have some ideas of how to tackle this but it will probably take some time to get them all working properly. For now I'm just going to support the OpenAI format since it is mostly the standard, but am looking at having a "reduced token" format as an option.

[markmghali]

Okay! Very interesting! Thank you for the explanation! I had no idea! Thank you for the work! Maybe keep it as a custom/ HACs integration

…

On Sat, Jun 1, 2024, 8:04 AM acon96 ***@***.***> wrote: Yeah I have a branch I'm working on to get support added. I should push it up to github once it's a bit more polished. There is a pretty significant amount of work to be done to refactor this integration to follow the new API patterns as well as refactor the dataset generation scripts so that I can train a new set of models that would work with the new APIs. The issue is that the APIs being added take a completely different mindset/methodology for prompting the models that is very different from how this integration currently prompts models. The new APIs that HA is introducing seem to be following the OpenAI function calling standards that I have explicitly avoided because they are overly verbose (they use giant JSON objects using lots of tokens) which is only useful if you have a giant GPU to run your model or lots of time to wait around for an answer. This is explicitly BAD for a real-time voice assistant that is running locally on lower-spec hardware (i.e. CPU). Most of the features in this integration are focused around reducing the number of tokens that need to be processed to ensure the fastest response time possible. I do have some ideas of how to tackle this but it will probably take some time to get them all working properly. — Reply to this email directly, view it on GitHub <#154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEENN4GAWD7Q5ZYBYU4FDA3ZFHBHTAVCNFSM6AAAAABIRMEXYCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMMRYGY3TS> . You are receiving this because you authored the thread.Message ID: ***@***.***>