Any way to use local models?

[Neltherion]

I'm real tired of OpenAI/Groq/... API keys and having to call their APIs over the net. Is there any way to spin up a local LLM/Vision Model and direct the queries to that?

[simularlyon]

Yes, you can use Ollama for local models.

[saa1605]

The easiest way in the current setup is to use vLLM (https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) to locally load and host a model on your machine. After this you can use the LMMEnginevLLM by passing the required engine_params and setting the vLLM_ENDPOINT_URL to the locally hosted model.

1. Export the vLLM_ENDPOINT_URL:

export vLLM_ENDPOINT_URL=YOUR_URL_ADDRESS

2. Specify engine params to use vllm before instantiating the GraphSearchAgent

engine_params = {'engine_type': 'vllm', 'model': 'YOUR_HOSTED_MODEL_NAME'}