Decouple model_context from AssistantAgent

[akurniawan]

Why are these changes needed?

This is to decouple model_context from AssistantAgent to provide flexibility in augmenting incoming message to the agent. This could be useful when we have a specific template for an agent that is a part of a GroupChat, discussion here: #4668

Usage example

class MyModelContext()
  .....
  ....
my_model_context = MyModelContext(...)
agent = AssistantAgent(model_client =(..), model_context=(my_model_context...))
team = RoundRobinGroupChat(participantants=[agent] ...)

Related issue number

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[akurniawan]

@akurniawan please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree

Copilot reviewed 1 out of 1 changed files in this pull request and generated no suggestions.

Comments skipped due to low confidence (1)

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py:317

• The error message 'No tools are available.' could be more descriptive. Consider changing it to 'No tools or handoff tools are available for execution.'

raise ValueError("No tools are available.")

[ekzhu]

@victordibia please take a look at this and see the implication to #4438 .

@akurniawan thanks for the PR, since this is very closely related to the aforementioned PR, please change this to DRAFT state and we will need to coordinate the work.

[victordibia]

thanks @akurniawan , @ekzhu, @jackgerrits
PR is looking good.

Current PR

From my understanding this PR does the following:

• adds model_context as a argument to AssistantAgent .. based on the ChatCompletionContext class
• uses this model_context when passed or adds and instantiates one based on UnboundedBufferedChatCompletionContext
• The expected usage/benefit is that at design time, the developer can pass in their own model_context object and control behaviors on how/what context is stored by overloading the add_message method e.g., transform the structure of the message before it is added, drop some messages, replace etc.

@akurniawan .. can you confirm the above is correct? Also, if possible can you update the PR description with an example usage ...e.g., below?

class MyModelContext()
  .....
  ....
my_model_context = MyModelContext(...)
agent = AssistantAgent(model_client =(..), model_context=(my_model_context...))
team = RoundRobinGroupChat(participantants=[agent] ...)

PR #4438 - Memory in AgentChat

#4438 focuses on

• adding adding a memoryargument toAssistantAgent`.
• At runtime, if a memory object is passed, effort is made to query it to update the exact context message just in time (RAG) by retrieving from the memory store. E.g., I pass in a memory argument with my preferences (e.g., I am vegan) and that is retrieved and added to a query like provide a meal plan. (stuff can be added/removed from memory external to the assistant agent resulting in dynamic behaviours during runtime)

My feeling is that there are differences - while model_context is focused fixed design time updates to all messages, memory is more towards just-in-time updates. I feel both can coexist

• model_context is used to transform all messages
• memory is used to append stuff relevant, each time the agent receives a message. Implementation would be to add retrieved memory messages to llm_messages

Thoughts welcome.

[ekzhu]

My feeling is that there can be two things:

1. memory, which has a method that generates or transforms a model context
2. model context, an object that contains the LLMMessage that will be sent to the model

I think that the current model context interface is not quite complete, because a model context should also include tools or tool schema -- something a memory module should also provide augmentation for.

In this regard, I think the model context object could be potentially transient -- the agent only create one on demand when model inference is performed.

[akurniawan]

@victordibia yes that's correct and I have updated the PR description as suggested

[ekzhu]

@akurniawan thanks.

@victordibia and I have discussed and we think this PR can be marked as Ready for Review.

[ekzhu]

@akurniawan I think this PR is almost at the finish line. I believe we just need to resolve the few remaining issues.

Would you like us to take over and get it done?

[ekzhu]

@akurniawan I am going to push some changes.

[akurniawan]

@akurniawan I think this PR is almost at the finish line. I believe we just need to resolve the few remaining issues.

Would you like us to take over and get it done?

Hey sorry was not available yesterday. I can complete the changes you suggested today.

[akurniawan]

it seems you have already changed it, thanks for the help!

[ekzhu]

You are welcome 😃

Join our office hours and discord to discuss more.

https://aka.ms/autogen-officehour

https://aka.ms/autogen-discord

[victordibia]

Looks good, thanks.

to add memory, we can do something like

...
# Inner messages.
        inner_messages: List[AgentEvent | ChatMessage] = [] 

        # Transform the model context for each "memory source"
        if memory and isinstance(memory, list):
            for m in memory: 
                self._model_context = await m.transform(self._model_context, transform=True) 
                

@ekzhu , what do you think?

[husseinmozannar]

We can either use a list or the ChatCompletionContext without anything breaking right?

[ekzhu]

Looks good, thanks.

to add memory, we can do something like

...

# Inner messages.

        inner_messages: List[AgentEvent | ChatMessage] = [] 



        # Transform the model context for each "memory source"

        if memory and isinstance(memory, list):

            for m in memory: 

                self._model_context = await m.transform(self._model_context, transform=True) 

                

@ekzhu , what do you think?

Yes. That's what I was thinking. The model context is essentially the input or the "working set" that model consumes.

[ekzhu]

We can either use a list or the ChatCompletionContext without anything breaking right?

Right now no one is using this new constructor argument so it will not be breaking change.

[ekzhu]

Looks good, thanks.

to add memory, we can do something like

...

# Inner messages.

        inner_messages: List[AgentEvent | ChatMessage] = [] 



        # Transform the model context for each "memory source"

        if memory and isinstance(memory, list):

            for m in memory: 

                self._model_context = await m.transform(self._model_context, transform=True) 

                

@ekzhu , what do you think?

Perhaps the memory can directly mutate the context. Right now the code snippet just shows direct update. We want be prepared for a scenario in which a remote model context hosted on a service. Eg OpenAI Assistant API's thread.

[victordibia]

Perhaps the memory can directly mutate the context. Right now the code snippet just shows direct update. We want be prepared for a scenario in which a remote model context hosted on a service. Eg OpenAI Assistant API's thread.

Makes sense ... we can discuss this on the memory PR once this is merged.
Looks ready to merge.