Surfacing errors in LangChain4j tools in dev mode in the dev UI?

[holly-cummins]

If an @Tool throws an exception during execution, the stack trace turns up in the quarkus console, but it can be masked by request and response logging. In my experience, it seems like the response logging seems to mostly to hit the log second, and especially if there's a few tools defined, it can be pretty verbose. So to even notice that there was a tool problem, a lot of scrolling is needed.

Here's an example, where the details don't matter, but you can see on the right side how far back I had to scroll to get to the stack trace:

I wonder if there's some value in surfacing the exception to the browser window? The model I'm thinking of is the normal dev mode exception handling, which is helpful to developers:

We couldn't use that exact behaviour in this case, because the server doesn't return a 500, it goes down a success path and returns 200. My first thought was that we'd detect the error case and render a split screen with the string response in one half and the stack trace in the other.

That doesn't really work, I don't think, because it assumes the client is a browser. If the client is an REST client that was trying to consume API output, giving it html would be a disaster. But maybe there's something that can be done. We could set the favicon, but browsers tend to cache that. We could set headers, but I don't know what header we could choose that a browser would do something interesting with.

At the very least, I think it would be safe to surface the errors in the main Dev UI? Perhaps something like this, where that row only shows if a bad thing happened during a tool invocation?

[geoand]

Very interesting usability issue indeed!

I am not sure what can be done about honestly... @jmartisk @phillip-kruger any ideas?

[phillip-kruger]

You can stream a dedicated log in the footer for tool errors. Then you can clean up the log in there as you wish, maybe make the stack collapsible

[holly-cummins]

Ah, my comment crossed with @phillip-kruger's! I was going to suggest that there's probably stuff that can be done in the in-Dev-UI log window to separate out the request/response logging (which can be super-verbose) from the rest, maybe by having two tabs.

Ideally, that would be part of the solutions along with something higher up the page.

[jmartisk]

You can stream a dedicated log in the footer for tool errors. Then you can clean up the log in there as you wish, maybe make the stack collapsible

+1 this would be cool

[jmartisk]

I'll have a shot at implementing the dedicated log if you don't mind

[jmartisk]

Ah, the footer log feature is from Quarkus 3.16, so we can't use it yet (we are based on 3.15), right @phillip-kruger?

[phillip-kruger]

Yes, but you can still do it in 3.15, it's just more work