Investigate a way to handle handle dangerous tools (do not rely on LLM)

[wizhippo]

I am trying to sort out the best way to handle dangerous tools and was thinking of agent that is aware of tools decorated with "@requires_confirmation_tool" and will not run them without confirmation gist example

The current agent has exit_conditions, but this exits too late, the tool is already called and there is no built in way to intercept before the tool is called by the LLM.

Perhaps this may fit in the toolinvoker, and the invoker could guard tools with a dangerous attribute and require some sort of intervention external to the LLM to allow invocation...

I have experimented with a standard tool that has a "comfirmed" parameter but I don't trust the LLM enough to not be creative and just set it regardless if the context indicates otherwise.

This is kind of like human in the loop but maybe not? Not sure if the definition of human in the loop dictates that the LLM is not responsible for invoking the tool.

I would love some discussion on how others have handled this and perhaps a solution could be added to haystack natively

[sebastianh-deepset]

I have a similar need for an Agent that uses tools requiring explicit human approval before execution - essentially, the Agent asking: “Do you approve calling this tool?”.

So far, these are the approaches that I've found in Haystack:

Human in the Loop Tool (from this cookbook):

• Appears to be intended as a fallback mechanism when the Agent decides it needs more context
• Enforcement seems to be mostly prompt-based rather than deterministic → Probably a deal breaker for production level
• Triggers the pipeline again with the human’s answer as a new query.

Agent Breakpoints / ToolBreakpoints:

• Deterministic: you can specify exactly which tools trigger a breakpoint, regardless of LLM behavior, which is great
• Currently more like a debugging mechanism than a true “Human in the Loop” control
• Multiple tools can be set to trigger breaks
• visit_count controls when the break happens - is there a way to trigger every time a specific tool is called?
• Currently you can't mix breakpoints and resuming from a snapshot after approval

This would be the desired behavior:

• A set of “permission-required” tools can be defined (or marked)
• Calling one of these tools always triggers a human-approval step that requires a user input (e.g. a boolean):
• True → approve and execute the tool
• False → skip the tool call
• Pipeline state up to the breakpoint is preserved, including which tool triggered it to continue there instead of re-running the pipeline with a new query
• The Boolean approval applies only to the current pipeline run and only for that specific trigger tool. If the same tool is called later in the session, it should again require approval
• Should not only work in Jupyter Notebooks or locally but also behind an API to be production ready -> Not only Haystack but also Hayhooks