Add OpenRoughter Provider Roughting?

[Lostsite]

OpenRoughter would allow user choose from multiple different providers for the same model. Sometimes people might don't want use the default provider choose by OpenRoughter (For example, for price/privacy reasons etc.). OpenRoughter do enable provider routing function ( https://openrouter.ai/docs/provider-routing ). If we can request to add this function into lobe-chat?

[arvinxx]

I think it's a diffcult work to add a UI for user to handle the provider routing params. So the answer is no.

[Lostsite]

Based on the document, the code should be very simple, like this

"provider": {
      "order": [
        "OpenAI",
        "Together"
      ]
    },

I means we can hide it from the GUI interface settings, and only put it as an options inside the .env file, like in model list, support:

id=displayName<maxToken:vision:fc:file:**provider1**,**provider2**>

This give more experienced user more control (Because on OpenRoughter current load balance settings, sometimes they would send your request to most expensive provider to balance the load if you don't set the provider routing). If they screw it up, they know how to undo it. While keep the Gui interface simple and avoid make the beginners confused.

[arvinxx]

It will be much easier to do if there is no GUI. If we do it, it should be another env like OPENROUTER_PROVIDER_ROUTE.

OPENROUTER_PROVIDER_ROUTE=OpenAI,Together,etc,...

the only change is to add an handlePayload in the OpenRouter provider runtime:

export const LobeOpenRouterAI = LobeOpenAICompatibleFactory({
  baseURL: 'https://openrouter.ai/api/v1',
  chatCompletion: {
    handlePayload: (payload) => {
      return {
        ...payload,
        stream: true,
        "provider": {  "order": env.OPENROUTER_PROVIDER_ROUTE.toArray },
      } 
    },
  },

the file is here: https://github.com/lobehub/lobe-chat/blob/main/src/libs/agent-runtime/openrouter/index.ts

[Lostsite]

Just a comment, if we put it into the model level, it could give people more control on the model based level. Since some provider doesn't support some model at all, some providers is more expansive on some model, cheaper on others, and some might get int8 on some model (I want to avoid) but doing great on others.

[arvinxx]

like this?

OPENROUTER_PROVIDER_ROUTE="llama3.1=OpenAI,Together;deepseek=DeepSeek,Together"

[Lostsite]

That would work. Thanks for considering it.