Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/output.md
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,12 @@ When the model calls other tools in parallel with an output tool, you can contro

The `'exhaustive'` strategy is useful when tools have important side effects (like logging, sending notifications, or updating metrics) that should always execute.

!!! warning "Streaming vs Sync Behavior Difference"
`run_stream()` behaves differently from `run()` and `run_sync()` when choosing the final result:

- **`run_stream()`**: The first called tool that **can** produce a final result (output or deferred) becomes the final result
- **`run()` / `run_sync()`**: The first **output** tool becomes the final result. If none are called, all **deferred** tools become the final result as `DeferredToolRequests`
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If none are called

The wording is like this, since we get an UnexpectedModelBehavior if both output and deferred tools are called, but none output tools are validated

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the deferred tools args are not validated

I guess it means we could check that if any deferred tools calls are present and tool_call_results is None, we can skip ignore invalid output tool calls, as we do in here:

yield _messages.FunctionToolCallEvent(call)
output_parts.append(e.tool_retry)
yield _messages.FunctionToolResultEvent(e.tool_retry)

But I'm not sure about it...
And IMHO, if do this at all, then it should be done in a separate PR


#### Native Output

Native Output mode uses a model's native "Structured Outputs" feature (aka "JSON Schema response format"), where the model is forced to only output text matching the provided JSON schema. Note that this is not supported by all models, and sometimes comes with restrictions. For example, Gemini cannot use tools at the same time as structured output, and attempting to do so will result in an error.
Expand Down
9 changes: 8 additions & 1 deletion tests/test_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -3291,7 +3291,14 @@ def deferred_tool(x: int) -> int: # pragma: no cover
)

def test_early_strategy_with_external_tool_call(self):
"""Test that early strategy handles external tool calls correctly."""
"""Test that early strategy handles external tool calls correctly.

Streaming and sync modes differ in how they choose the final result:
- Streaming: First tool call (in response order) that can produce a final result (output or deferred)
- Sync: First output tool (if none called, all deferred tools become final result)

See https://github.com/pydantic/pydantic-ai/issues/3636#issuecomment-3618800480 for details.
"""
tool_called: list[str] = []

def return_model(_: list[ModelMessage], info: AgentInfo) -> ModelResponse:
Expand Down
8 changes: 5 additions & 3 deletions tests/test_streaming.py
Original file line number Diff line number Diff line change
Expand Up @@ -1110,9 +1110,11 @@ def deferred_tool(x: int) -> int: # pragma: no cover
async def test_early_strategy_with_external_tool_call(self):
"""Test that early strategy handles external tool calls correctly.

Streaming mode expects the first output tool call to be the final result,
and has different behavior from sync mode in this regard.
See https://github.com/pydantic/pydantic-ai/issues/3636 for details.
Streaming and sync modes differ in how they choose the final result:
- Streaming: First tool call (in response order) that can produce a final result (output or deferred)
- Sync: First output tool (if none called, all deferred tools become final result)

See https://github.com/pydantic/pydantic-ai/issues/3636#issuecomment-3618800480 for details.
"""
tool_called: list[str] = []

Expand Down