Made the message role of ReAct observation configurable

[jamesljlster]

Description

This pull request made the message role of ReAct observation configurable.

Original title: Changed the message role of ReAct observation to tool
The purpose of this pull request was changed after discussing it with @logan-markewich , please refer to the conversation below.

I am developing a chatbot with the ReAct agent. Sometimes the chatbot gives strange responses to the user. After observing with tracer, I believe the problem is related to the inappropriate message role of tool message (observation), making the ReAct agent chat itself:

After setting the observation message role to tool, the stability got improved:

Related pull request: #17273 (the above test was made after #17273 was merged)
Test model: https://ollama.com/library/qwen2.5
Tracer: https://docs.arize.com/phoenix/tracing/integrations-tracing/llamaindex
Test code:

from llama_index.core import Settings
from llama_index.core.agent import ReActAgent
from llama_index.core.base.llms.types import ChatMessage, MessageRole
from llama_index.core.bridge.pydantic import Field
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai_like import OpenAILike

from phoenix.otel import register
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor


# Setup LLM
Settings.llm = OpenAILike(  # type: ignore
    api_base="http://localhost:11434/v1",
    api_key="dummy",
    model="qwen2.5",
    is_chat_model=True,
    timeout=300,
)


# Define tools
def get_weather(
    location: str = Field(
        description="A city name and state, formatted like '<name>, <state>'"
    ),
) -> str:
    """Useful for getting the weather for a given location."""
    return "Sunny"


class Greeter:
    """Useful for handling the general messages from users."""

    __name__ = "greeter"
    _memory: ChatMemoryBuffer

    def __init__(self, memory: ChatMemoryBuffer):
        self._memory = memory

    def __call__(
        self, message: str = Field(description="The message from the user.")
    ) -> str:
        response = Settings.llm.chat(
            [
                ChatMessage(
                    role=MessageRole.SYSTEM,
                    content=(
                        "You are a helpful assistant responsible for answering user "
                        "about the weather associated questions. You should focus on "
                        "your responsibility, don't answer the irrelevent questions."
                    ),
                ),
                *self._memory.get(),
                ChatMessage(role=MessageRole.USER, content=message),
            ]
        )

        assert response.message.content is not None
        return response.message.content


# Setup ReAct agent
memory = ChatMemoryBuffer.from_defaults()
agent = ReActAgent.from_tools(
    [
        FunctionTool.from_defaults(get_weather),
        FunctionTool.from_defaults(Greeter(memory)),
    ],
    llm=Settings.llm,
    memory=memory,
    verbose=True,
)

# Setup tracer
tracer_provider = register(
    project_name="ReAct Agent Experiment",
    endpoint="http://localhost:6006/v1/traces",
)
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)

# Chat with the agent
while True:
    response = agent.chat(input("User Input: "))
    print(response)

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• I added new unit tests to cover this change
• I believe this change is already covered by existing unit tests

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[logan-markewich]

This assumes that the LLM you are using supports a tool role though? If your LLM supports a tool role already, you should be using FunctionCallingAgent instead of react anyways? 🤔

[jamesljlster]

I'm not sure if a model supports tool role, it supports function/tool calling too. Things are a bit complicated in my situation. I use Llama 3.1 and 3.3 hosted by vLLM in my company. The both supports the tool role, but I can't make vLLM produces the correct tool calling responses. The problem might be related to the tool parser plugin, and I currently have no idea how to fix it.

Anyway, I think it's better to set the message role to the appropriate one. If legacy models don't support the tool role, should I make it configurable?

[logan-markewich]

@jamesljlster the vllm issue sounds like the real issue. Assuming vLLM is launched in openai-compatible server mode, it should be straightforward

pip install llama-index-llms-opeani-like

from llama_index.llms.openai_like import OpenAILike
from llama_index.core.agent import FunctionCallingAgent

llm = OpenAILike(model="some model", api_key="fake", api_base="http://localhost:800/v1", is_chat_model=True, is_function_calling_model=True)

agent = FunctionCallingAgent.from_tools(tools, llm=llm)

[jamesljlster]

@logan-markewich

I've launched vLLM in openai-compatible server mode with docker. The chatting function works very well but tool calling just doesn't work. It always produces the tool calls in message.content field instead of the correct message.tool_calls field in the API response.
I'm still investigating how to fix it. In the meantime, I want a stable agent so my chatbot can work properly.

I believe that the ReAct agent is designed to work with any LLMs with reasoning ability. When a new model arrives, the inference server projects may not provide the full functionality for the model immediately. (For example, chatting works but tool calling is not yet ready). At this moment, the ReAct agent may be the best choice.
Also, I don't have to be worried about having no suitable agents to use when the tool calling feature is not yet ready or broken.

I may add a default argument observation_role to MessageRole.USER for ReActChatFormatter. The default behavior stays unchanged while making it configurable via the initialization method.
What do you think? I wish to know your thoughts.

[logan-markewich]

@jamesljlster yea sure, I'd rather it be configurable.

[jamesljlster]

@logan-markewich I've made the message role of observations configurable.

I would like to let the observation role parameter documented in https://docs.llamaindex.ai/en/stable/api_reference/agent/react/#llama_index.core.agent.react.ReActChatFormatter, so I use Pydantic Field to annotate it.
I want to update the current ReAct document to include this change too: https://docs.llamaindex.ai/en/stable/examples/agent/react_agent/

However, I met a problem building the documentation:

The error is not related to my changes. How can I resolve it?

[jamesljlster]

Nevermind! I would use the latest release to develop the documentation. The problem does not appear in the latest release code base.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[jamesljlster]

@logan-markewich I think this pull request is finished. Would you review it for me?

[jamesljlster]

@logan-markewich May I request your feedback on my latest changes?