ReAct agent returns error messages as "successful" answers #1309
Description
Is this a new feature, an improvement, or a change to existing functionality?
Improvement
How would you describe the priority of this feature request
High
Please provide a clear description of problem this feature solves
ReAct agent returns error messages as "successful" answers
Description
The ReAct agent silently swallows parsing errors and returns them as "successful" answers. There is no exception raised, no error flag set, and no way to distinguish a failed run from a successful one without parsing the output text yourself.
This means callers must implement their own text-based error detection (e.g., checking if the answer contains "Invalid Format") to know if the agent actually succeeded - defeating the purpose of having a structured API.
Reproduction
The bug is intermittent and depends on how the LLM formats its response.
#!/usr/bin/env python3
"""Minimal reproducer: NAT ReAct agent returns error messages as 'successful' answers."""
import asyncio
import tempfile
from nat.builder.function import FunctionGroup
from nat.builder.framework_enum import LLMFrameworkEnum
from nat.cli.register_workflow import register_function_group
from nat.data_models.function import FunctionGroupBaseConfig
from nat.runtime.loader import load_workflow
# Register a simple tool
class SimpleToolConfig(FunctionGroupBaseConfig, name="simple_tool"):
pass
@register_function_group(config_type=SimpleToolConfig, framework_wrappers=[LLMFrameworkEnum.LANGCHAIN])
async def simple_tool(config: SimpleToolConfig, builder):
group = FunctionGroup(config=config)
async def add(a: int, b: int) -> str:
"""Add two numbers."""
return f"{a} + {b} = {a + b}"
group.add_function(name="add", fn=add, description=add.__doc__)
yield group
CONFIG = """
function_groups:
simple_tool:
_type: simple_tool
llms:
nemotron:
_type: azure_openai
api_key: $NV_INFER
model_name: nvidia/nvidia/Nemotron-3-Nano-30B-A3B
azure_deployment: nvidia/nvidia/Nemotron-3-Nano-30B-A3B
azure_endpoint: https://inference-api.nvidia.com
api_version: latest
workflow:
_type: react_agent
llm_name: nemotron
tool_names: [simple_tool]
"""
async def main():
with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
f.write(CONFIG)
config_path = f.name
async with load_workflow(config_path) as workflow:
async with workflow.run("What is 2 + 3?") as runner:
answer = await runner.result()
print(f"\n{'='*60}")
print(f"Answer: {answer}")
print(f"{'='*60}")
if "Invalid Format" in answer or "Missing 'Action" in answer:
print("BUG: Error message returned as 'successful' answer!")
else:
print("OK (or try again - bug depends on LLM output format)")
if __name__ == "__main__":
asyncio.run(main())Actual Behavior
The agent returns internal error messages like "Invalid Format" or "Missing 'Action..." as the final "successful" answer. There is no programmatic way to detect this failure:
- No exception is raised
- No error status/flag is returned
- The only way to detect failure is to parse the answer string for known error substrings
Root Cause
The bug is in agent.py:232-235 in the agent_node method.
When parsing fails after max retries, instead of raising an exception, the code silently converts the error message into a "successful" response:
# Lines 232-235: Error message becomes the answer
combined_content = str(ex.observation) + '\n' + str(output_message.content)
output_message.content = combined_content
state.messages += [output_message]
return state # Returns normally - caller sees this as successThe ex.observation contains error strings like "Invalid Format: Missing 'Action:' after 'Thought:'" (defined in output_parser.py:26-29).
Describe your ideal solution
The agent should either:
- Raise an exception when parsing fails after max retries, or
- Return a structured result with an error status that callers can check programmatically
Additional context
- The bug is non-deterministic and depends on the LLM's output format
- Observed with Nemotron model but likely affects other models that don't strictly follow the ReAct format
Code of Conduct
- I agree to follow this project's Code of Conduct
- I have searched the open feature requests and have found no duplicates for this feature request