-
Notifications
You must be signed in to change notification settings - Fork 214
Open
Description
AgentScope-Java is an open-source project. To involve a broader community, we recommend asking your questions in English.
Description
Currently, the five primary implementations of ChatModelBase (OpenAIChatModel, AnthropicChatModel, DashScopeChatModel, GeminiChatModel, and OllamaChatModel) exhibit inconsistent behavior when handling non-streaming calls.
The Core Issue
In non-streaming mode (stream=false), different implementation classes exhibit different blocking behaviors towards the calling thread:
| Implementation | Non-streaming |
|---|---|
| OpenAIChatModel | Blocking |
| AnthropicChatModel | Non-blocking |
| DashScopeChatModel | Blocking |
| GeminiChatModel | Non-blocking |
| OllamaChatModel | Blocking |
Why This Needs to be Addressed
- Semantic Inconsistency: The same client code behaves differently depending on the underlying model. This leads to unpredictable application performance and inconsistent API semantics.
- Developer Experience: Users expect a unified Reactive interface (returning
Flux/Mono) to follow consistent non-blocking patterns. - Maintenance Complexity: It increases the cognitive load for developers integrating multiple models.
Proposal: Standardize on Non-blocking Mode
I propose that all ChatModelBase implementations should be updated to ensure that non-streaming calls do not block the calling thread.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
Backlog