Add default chat model to support dev UI chat?

[holly-cummins]

I got in trouble in a demo recently, because I hadn't noticed when practicing that the chat feature in the dev UI is only available if the application contains a chat model. I wanted to show the flow where you can interact with the model before creating the application, which I'd seen other people show, but it didn't work for me. This is documented, but it caught me by surprise.

Two things make this extra-surprising, IMO:

• "Adding a chat model to the application" can be pretty implicit. While it can be done with explicit config in application.properties, it's sufficient to add a @RegisterAIService annotation somewhere. If we can do some default configuration to support @RegisterAIService, would it be a bad thing to do similar defaulting to support the dev UI?
• Other interaction modes in the dev UI, like creating images, work before an application is created. Images only work if the right extension is present, so the rule is different for images and chat; images we will enable by default if the right provider is present, but chat we will only enable with explicit configuration or once an application has been created

Obviously, we should design for the best experience for real users, not for demos. But I wonder if other people might be surprised by the current behaviour, and if extending what we default-configure might help others?

[jmartisk]

Hmm, good point. The main difference is that the Image page doesn't use an ImageModel injected from CDI, but instead, it lets the user configure all its parameters in the page, and then upon submitting, an ImageModel instance is dynamically generated from these parameters.

For chat models, we could potentially do the same, but we would probably have to supply a similar form to specify all parameters, and the problem with this is that each provider type has different parameters, so the page would no longer be the same for all possible providers, it would probably have to move to each provider extension separately (the image page is currently only in the OpenAI extension).

Also, if we did this, I'm not sure how exactly we would support users who just want to test the already configured model from CDI, the UI and the related logic might get very clunky for this (it already is quite a mess, to be honest, and extremely hard to test, because we don't have any proper infrastructure capable of testing the Dev UI pages, only the related JSON-RPC services)...

[jmartisk]

So yes, it would probably be possible, I'm just scared it would be much harder to implement and maintain that one might think

[jmartisk]

Even if we just went with an 'easy' solution where we dynamically instantiate a model with default parameters, it would still have to be tied to each provider separately, because you need to use the concrete provider's ChatModel builder. For Images it is easy, because we only support OpenAI atm.

[holly-cummins]

Hmm, tricky. If it's a case of lots of internal complexity in order to give an external experience that looks trivially simple, that's kind of our reason for being, as developers. But of course there's zillions of things to do, so if it's a pretty marginal user benefit, maybe it's not worth it. It's probably fairly rare that people would try the dev ui before adding a @RegisterAIService annotation somewhere in their codebase.

[holly-cummins]

Thinking about it more, I think I agree a form on the UI wouldn't be ideal, both because it means more clicking for users, and because of the risks @jmartisk identified of conflicts between the UI model and the CDI model. What envisaged was more 'dev UI will behave the same as the app,' ie 'you don't have to specify a model and there's a sensible default.'