AI Agents for local GPUs, by Voyage Robotics.
Originally built for a hackathon in 24 hours.
Find a 7B param multi-modal LLaVa x Mistral model in HuggingFace. You can also build models yourself using the llama.cpp library.
This is a standalone package. You need to install the llama_cpp python bindings as well to use voyage_agents.
See the llama-cpp-python install instructions in the README as the process varies based on your machine configuration, e.g. whether to use CUDA or not.
pip install llama-cpp-python
pip install voyage_agents
Imports and model creation.
from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llava16ChatHandler
from voyage_agents.core import LlamaManager
system_prompt = "You are an intelligent robotic assistant that is navigating autonomously in an environment, attempting to complete a given task, and assisting a user with queries."
model_path = "data/llava-v1.6-mistral-7b/llava-v1.6-mistral-7b-8B-F32.gguf"
clip_model_path="data/llava-v1.6-mistral-7b/mmproj.bin"
chat_handler = Llava16ChatHandler(clip_model_path=clip_model_path)
llama = Llama(
model_path=model_path,
chat_handler=chat_handler,
n_gpu_layers=20, # Uncomment to use GPU acceleration
n_ctx=4092, # Uncomment to increase the context window
seed=1337, # Uncomment to set a specific seed
)
manager = LlamaManager(llama, temperature=0)Calling a tool based on a prompt, and returning the result in structured form.
from voyage_agents.agents import ToolCaller
from voyage_agents.tool import SearchObjectsTool, PickObjectTool, ChangeTaskTool
tool_caller = ToolCaller(manager, [SearchObjectsTool(), PickObjectTool(), ChangeTaskTool()], system_prompt)
# easy task
print(tool_caller.call("have you found the object you are looking for yet?"))Answering a question with the use of tools.
from voyage_agents.agents import QuestionAnswerer
from voyage_agents.tool import SearchObjectsTool, PickObjectTool, ChangeTaskTool
agent = QuestionAnswerer(manager, [SearchObjectsTool(), PickObjectTool(), ChangeTaskTool()], system_prompt)
# easy task
answer = agent.run("how many tables has the robot seen so far?")
print(answer)See the tools package for examples on how you'd implement your own tools.