Skip to content

Conversation

@Danipulok
Copy link
Contributor

Closes #3624

@Danipulok
Copy link
Contributor Author

When I was working on tests, I discovered that agent.run_stream + result.stream_text(delta=False) (default behavior) does not call TextOutput(my_tool)

It seems this is what was discussed in #3393 (comment), so it does not deserve a new issue?

Here are the tests that demonstrate bugged behavior: link

Should I include the failing streaming test with xfail or don't commit it at all?

@Danipulok
Copy link
Contributor Author

About tests: I created a new class designed to test partial_output and separated tests into tests for sync and streaming mode.
The tests were not deleted, just moved and formatted to look alike

Here are the changes:

test_output_validator_partial_sync

before: test_agent.py::test_output_validator_partial_sync

def test_output_validator_partial_sync():
"""Test that output validators receive correct value for `partial_output` in sync mode."""
call_log: list[tuple[str, bool]] = []
agent = Agent[None, str](TestModel(custom_output_text='test output'))
@agent.output_validator
def validate_output(ctx: RunContext[None], output: str) -> str:
call_log.append((output, ctx.partial_output))
return output
result = agent.run_sync('Hello')
assert result.output == 'test output'
assert call_log == snapshot([('test output', False)])

after: test_agent.py::TestPartialOutput::test_output_validator_text
https://github.com/Danipulok/pydantic-ai/blob/d08191ee0e84ea2ae54fd4c28e8d5076986c281e/tests/test_agent.py#L370-L387

test_output_validator_partial_stream_text
before: test_agent.py::test_output_validator_partial_stream_text

async def test_output_validator_partial_stream_text():
"""Test that output validators receive correct value for `partial_output` when using stream_text()."""
call_log: list[tuple[str, bool]] = []
async def stream_text(messages: list[ModelMessage], info: AgentInfo) -> AsyncIterator[str]:
for chunk in ['Hello', ' ', 'world', '!']:
yield chunk
agent = Agent(FunctionModel(stream_function=stream_text))
@agent.output_validator
def validate_output(ctx: RunContext[None], output: str) -> str:
call_log.append((output, ctx.partial_output))
return output
async with agent.run_stream('Hello') as result:
text_parts = []
async for chunk in result.stream_text(debounce_by=None):
text_parts.append(chunk)
assert text_parts[-1] == 'Hello world!'
assert call_log == snapshot(
[
('Hello', True),
('Hello ', True),
('Hello world', True),
('Hello world!', True),
('Hello world!', False),
]
)

after: test_streaming.py::TestPartialOutput::test_output_validator_text
https://github.com/Danipulok/pydantic-ai/blob/d08191ee0e84ea2ae54fd4c28e8d5076986c281e/tests/test_streaming.py#L762-L789

test_output_validator_partial_stream_output

before: test_agent.py::test_output_validator_partial_stream_output

async def test_output_validator_partial_stream_output():
"""Test that output validators receive correct value for `partial_output` when using stream_output()."""
call_log: list[tuple[Foo, bool]] = []
async def stream_model(messages: list[ModelMessage], info: AgentInfo) -> AsyncIterator[DeltaToolCalls]:
assert info.output_tools is not None
yield {0: DeltaToolCall(name=info.output_tools[0].name, json_args='{"a": 42')}
yield {0: DeltaToolCall(json_args=', "b": "f')}
yield {0: DeltaToolCall(json_args='oo"}')}
agent = Agent(FunctionModel(stream_function=stream_model), output_type=Foo)
@agent.output_validator
def validate_output(ctx: RunContext[None], output: Foo) -> Foo:
call_log.append((output, ctx.partial_output))
return output
async with agent.run_stream('Hello') as result:
outputs = [output async for output in result.stream_output(debounce_by=None)]
assert outputs[-1] == Foo(a=42, b='foo')
assert call_log == snapshot(
[
(Foo(a=42, b='f'), True),
(Foo(a=42, b='foo'), True),
(Foo(a=42, b='foo'), False),
]
)

after: test_streaming.py::TestPartialOutput::test_output_validator_structured
https://github.com/Danipulok/pydantic-ai/blob/d08191ee0e84ea2ae54fd4c28e8d5076986c281e/tests/test_streaming.py#L791-L818

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Output functions are called twice or more times in streaming mode

1 participant