[RFC] 052 - Cloudflare Workers AI Support

[BrandonStudio]

Add Cloudflare Workers AI provider.

Cloudflare Workers AI now supports following types of models:

• Text Generation
• Image-to-Text
• Automatic Speech Recognition
• Translation
• Text-to-Image
• Text Embeddings
• Image Classification
• Object Detection
• Summarization
• Text Classification

Where the first 5 types of models can be used in LobeChat.

• Use text generation models as chat models.
• Use Image-to-Text models as vision chat models.
• Use automatic speech recognition models as speech recognizer.
• Use translation models as translator (system assistant).
• Similar to DALL-E, add built-in plugin for text-to-image models.
• Use embedding models to help build knowledge base.

Required configurations

• Account ID
  This can be replaced with endpoint url.
• Authorization
  This can be either Bearer token (Authorization: Bearer {token}) or API key (X-Auth-Key: {key}).
• Endpoint URL
  This allows user to use AI Gateway or their own workers, which may be better than account ID.

Chat model capability

• ✔ All models support stream.
• ✖ Few models (one for now) support function calling.

Compatibility

• Although Cloudflare Workers AI is compatible with OpenAI API endpoints, it does not support function calling there.
• Users may have created a worker to reversely proxy AI requests. If this use case is permitted, authorization config could be optional.

[arvinxx]

there is a PR for it : #2966

[BrandonStudio]

I see it now. It would be great, but

1. I think that PR uses OpenAI style API, which does not support function calling.
2. I think that PR does not support both bearer auth and api token.
3. It does not supports a custom endpoint.

[sxjeru]

Thank you for your proposal. But sorry, due to my technical limitations, I only consider implementing basic text generation.
I hope other contributors can complete and improve other functions.

[BrandonStudio]

Do you think we should support custom endpoint? What do you think of the difference between using account ID and custom endpoint?

[sxjeru]

Custom endpoints can be considered. But I don't think there will be many people using it.

[BrandonStudio]

@arvinxx How to handle an event-stream response?
Streams from Cloudflare Workers AI is like:

data: {"response":"你好","p":"abcdefghijklmnop"}
data: {"response":"！","p":"abcdefghijklmnopqrstuvwxyz012345"}
data: {"response":"很高兴","p":"abcdefghijklmnopqrstuv"}
data: {"response":"为你","p":"abcdefghijklmnopqrstuvwxyz0123456789abcdefghij"}
data: {"response":"提供","p":"abcdefghijklmnopqrstuvwxyz0123456789abcdefghijk"}
data: {"response":"帮助","p":"abc"}
data: {"response":"。","p":"a"}
data: {"response":"","p":"abcdefghij"}
data: [DONE]

Parsed:

你好！很高兴为你提供帮助。

And tool call response is like

data: {"response":"<","p":"abcdefghijk"}
data: {"response":"tool","p":"ab"}
data: {"response":"_","p":"abcdefghijklmnopqrstuvwxyz0123456789abc"}
data: {"response":"call","p":"abcdefghijklmn"}
data: {"response":">","p":"abcdefghijklmnopqrstuvwxyz0123456789ab"}
data: {"response":"\n","p":"abcdefghijklmnopqrstuvwxyz0123456"}
data: {"response":"{'","p":"abcdefghijklmnopqrstuvwxyz0123456789ab"}
data: {"response":"arguments","p":"abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklm"}
data: {"response":"':","p":"abcdefghijklmnop"}
data: {"response":" {'","p":"abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklm"}
data: {"response":"username","p":"abcdefghijklmnop"}
data: {"response":"':","p":"abcdefghijklmnopqrstuv"}
data: {"response":" '","p":"abcdefghijklmnopqrstuvwxy"}
data: {"response":"cloud","p":"abcdefghijklmnopqrstuvwxyz0123456"}
data: {"response":"fl","p":"abcdefghijklmnopqrstuvwxyz012345"}
data: {"response":"are","p":"abcdefghijklmnopqrstuvwxyz012345678"}
data: {"response":"'},","p":"abcdefghijklmnopqrstuvwxyz0123"}
data: {"response":" '","p":"abcdefghijklmnopqrstuvwxyz0123456789abcd"}
data: {"response":"name","p":"abcdefghijkl"}
data: {"response":"':","p":"abcdefghijklmnopqrstuvwxy"}
data: {"response":" '","p":"abcdefghijklmno"}
data: {"response":"get","p":"abcdefghijklmnopqrstuvwxyz0"}
data: {"response":"G","p":"abcdefghijklmnopqrstuvwxyz01234"}
data: {"response":"ithub","p":"ab"}
data: {"response":"User","p":"abcdefghijklmnopqrs"}
data: {"response":"'}","p":"abcdefghijklmnopqrstuvwxyz012"}
data: {"response":"\n"}
data: {"response":"</","p":"abcdefghijk"}
data: {"response":"tool","p":"abcdefghijklmnopqrstuvwxyz0123456789abcdefg"}
data: {"response":"_","p":"abcdefghijklmnopq"}
data: {"response":"call","p":"abcdefghijklmnopqrstuvwxyz0123456789abcdef"}
data: {"response":">","p":"abcdefghijklmnopqrstuvwxyz0123456789a"}
data: {"response":"<|im_end|>","p":"abcdefghijklmnopqrstuvwxyz012345678"}
data: [DONE]

Parsed:

<tool_call>\n{'arguments': {'username': 'cloudflare'}, 'name': 'getGithubUser'}\n</tool_call><|im_end|>

The stream response of a tool call is not a JSON.
However, non-stream response is more readable:

{"result":{"response":null,"tool_calls":[{"arguments":{"username":"cloudflare"},"name":"getGithubUser"}]},"success":true,"errors":[],"messages":[]}

[arvinxx]

here is an example of handle stream response: #2794

[arvinxx]

@BrandonStudio I will write a post to talk about streaming tools calling these day. Maybe it will help.

[BrandonStudio]

By the way, how to integrate dynamic model list? Do we have a refresh delegate by which user will get up-to-date model list at front end? If we do, where should I set the handler?
As at BrandonStudio#1/index.ts#L163, I have a getModels method. How could I integrate it to the project?

[arvinxx]

There should be a param to auto fetch models when app init. But currently we don't implement it. If implemented, it should be as easy as to add a simple field like autoFetchModelList: true to ModelCard.

[BrandonStudio]

I saw models method defination in LobeRuntimeAI, and OpenAICompatibleFactory has already implemented it.
But when I rename getModels to models, although the response is in correct format, and data in onSuccess does appear to be an array, the pull-down list keeps unchanged.

I'm confused now. Do you mean that this delegate is not yet used to update the list?

[BrandonStudio]

I know now. I should add configuration in defaultModelProviderList closure.