feat: ai-proxy plugin #11499

shreemaan-abhishek · 2024-08-15T02:01:11Z

Description

Implement the AI-Proxy Plugin.

Checklist

I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

apisix/plugins/ai-proxy.lua

apisix/plugins/ai-proxy/drivers/openai.lua

apisix/plugins/ai-proxy.lua

bzp2010

no more comments now ✨

bzp2010 · 2024-09-10T01:14:25Z

apisix/init.lua

@@ -726,6 +726,7 @@ function _M.http_access_phase()
 end


+


If there are no substantive changes, please do not change the code style.

bzp2010 · 2024-09-10T01:31:21Z

t/plugin/ai-proxy.t

+                    if header_auth == "Bearer token" or query_auth == "apikey" then
+                        ngx.req.read_body()
+                        local body, err = ngx.req.get_body_data()
+                        local esc = body:gsub('"\\\""', '\"')


Is this still necessary? Just a confirmation that if it is then no changes are needed.

kayx23 · 2024-09-10T01:37:55Z

docs/en/latest/plugins/ai-proxy.md

+  }'
+```
+
+Since `passthrough` is not enabled upstream node can be any arbitrary value because it won't be contacted.


design question: why do we need the parameter passthrough? is it meant to be a toggle?

is this sentence needed here?

design question: why do we need the parameter passthrough? is it meant to be a toggle?

yes, kinda. when it's enabled, the response from LLM will be set as the request body to the upstream.

is this sentence needed here?

I think it's important to let users know, instead to let them be puzzled about it and then figure out later.

docs/en/latest/plugins/ai-proxy.md

kayx23 · 2024-09-10T01:50:47Z

docs/en/latest/config.json

@@ -96,7 +96,8 @@
            "plugins/fault-injection",
            "plugins/mocking",
            "plugins/degraphql",
-            "plugins/body-transformer"
+            "plugins/body-transformer",
+            "plugins/ai-proxy"


The recently added ai-* plugins should probably all go into a new category. Can be amended in a different PR.

Co-authored-by: Traky Deng <[email protected]>

zhoujiexiong · 2024-09-11T02:47:19Z

apisix/core/request.lua

@@ -334,6 +335,21 @@ function _M.get_body(max_size, ctx)
 end


+function _M.get_request_body_table()


Decode body according content-type or rename the method, e.g. get_json_request_body_table?

zhoujiexiong · 2024-09-11T02:50:34Z

apisix/plugins/ai-proxy.lua

+        return 400, "unsupported content-type: " .. ct
+    end
+
+    local request_table, err = core.request.get_request_body_table()


new naming style?

request_table, count_num, flag_boolean?

I don't understand what you mean.

Although adding a suffix makes it more readable, it is not the style used in the project.

oh I got it now 😂 . request_table makes it more readable and it can be understood that it's not a string.

zhoujiexiong · 2024-09-11T02:55:11Z

apisix/plugins/ai-proxy.lua

+        return
+    end
+
+    if core.table.try_read_attr(conf, "model", "options", "stream") then


Suggested change

if core.table.try_read_attr(conf, "model", "options", "stream") then

if request_table.stream then

zhoujiexiong · 2024-09-11T02:58:08Z

apisix/plugins/ai-proxy.lua

+    local res, err, httpc = ai_driver.request(conf, request_table, ctx)
+    if not res then
+        core.log.error("failed to send request to AI service: ", err)
+        return 500


use the predefined Ngx constants?

ngx.HTTP_NOT_ACCEPTABLE = 406
ngx.HTTP_REQUEST_TIMEOUT = 408
ngx.HTTP_CONFLICT = 409
ngx.HTTP_GONE = 410
ngx.HTTP_UPGRADE_REQUIRED = 426
ngx.HTTP_TOO_MANY_REQUESTS = 429
ngx.HTTP_CLOSE = 444
ngx.HTTP_ILLEGAL = 451
ngx.HTTP_INTERNAL_SERVER_ERROR = 500
ngx.HTTP_METHOD_NOT_IMPLEMENTED = 501
ngx.HTTP_BAD_GATEWAY = 502
ngx.HTTP_SERVICE_UNAVAILABLE = 503
ngx.HTTP_GATEWAY_TIMEOUT = 504
ngx.HTTP_VERSION_NOT_SUPPORTED = 505
ngx.HTTP_INSUFFICIENT_STORAGE = 507

zhoujiexiong · 2024-09-11T02:58:47Z

apisix/plugins/ai-proxy.lua

+    local body_reader = res.body_reader
+    if not body_reader then
+        core.log.error("LLM sent no response body")
+        return 500


zhoujiexiong · 2024-09-11T03:04:34Z

apisix/plugins/ai-proxy/drivers/openai.lua

+    end
+
+    local query_params = ""
+    if conf.auth.query and type(conf.auth.query) == "table" then


Suggested change

if conf.auth.query and type(conf.auth.query) == "table" then

if type(conf.auth.query) == "table" then

zhoujiexiong · 2024-09-11T03:06:08Z

apisix/plugins/ai-proxy/drivers/openai.lua

+        end
+    end
+
+    local path = (parsed_url.path or "/v1/chat/completions") ..  query_params


Suggested change

local path = (parsed_url.path or "/v1/chat/completions") .. query_params

local path = (parsed_url.path or DEFAULT_URI) .. query_params

zhoujiexiong · 2024-09-11T03:24:02Z

apisix/plugins/ai-proxy/drivers/openai.lua

+
+    local query_params = ""
+    if conf.auth.query and type(conf.auth.query) == "table" then
+        query_params = core.string.encode_args(conf.auth.query)


FYI, from lua-resty-http, params.query already support table type

local function _format_request(self, params) local version = params.version local headers = params.headers or {} local query = params.query or "" if type(query) == "table" then query = "?" .. ngx_encode_args(query) elseif query ~= "" and str_sub(query, 1, 1) ~= "?" then query = "?" .. query end -- Initialize request local req = { str_upper(params.method), " ", self.path_prefix or "", params.path, query, HTTP[version], -- Pre-allocate slots for minimum headers and carriage return. "", "", "", } local c = 7 -- req table index it's faster to do this inline vs table.insert -- Append headers for key, values in pairs(headers) do key = tostring(key) if type(values) == "table" then for _, value in pairs(values) do req[c] = key .. ": " .. tostring(value) .. "\r\n" c = c + 1 end else req[c] = key .. ": " .. tostring(values) .. "\r\n" c = c + 1 end end -- Close headers req[c] = "\r\n" return tbl_concat(req) end

zhoujiexiong · 2024-09-11T03:32:33Z

t/sse_server_example/main.go

+		fmt.Fprintf(w, "data: %s\n\n", time.Now().Format(time.RFC3339))
+
+		// Flush the data immediately to the client
+		if f, ok := w.(http.Flusher); ok {


do type assertion outside the loop? :D

zhoujiexiong · 2024-09-11T04:50:43Z

apisix/plugins/ai-proxy/drivers/openai.lua

+
+    local res, err = httpc:request(params)
+    if not res then
+        return 500, "failed to send request to LLM server: " .. err


Suggested change

return 500, "failed to send request to LLM server: " .. err

return internal_server_error, "failed to send request to LLM server: " .. err

zhoujiexiong · 2024-09-11T04:53:07Z

t/sse_server_example/main.go

+		fmt.Fprintf(w, "[ERROR]")
+		return
+	}
+	// A simple loop that sends a message every 2 seconds


Suggested change

// A simple loop that sends a message every 2 seconds

// A simple loop that sends a message every 500ms

Parameterize it?

feat: ai proxy plugin

b98a48f

shreemaan-abhishek force-pushed the ai-proxy branch from f5d902f to b98a48f Compare August 16, 2024 04:17

shreemaan-abhishek added 13 commits August 16, 2024 10:06

remove subrequest

8188ae4

fix diff test

7b83b3a

long line fix

97cafa5

completions typo in consts

35b1787

license

e18caef

plugins.t fix

28f06ae

handle empty req body problem

82f9692

auth schema fix

0577e8e

scheme and method

e5f00f7

auth and model.name required

c307b04

scheme in lua code forgot to commit

ef4cf84

tests

4bf6bd2

lint

42adfd1

shreemaan-abhishek marked this pull request as ready for review August 19, 2024 05:11

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request plugin labels Aug 19, 2024

shreemaan-abhishek added 3 commits August 20, 2024 11:53

add docs

0af00ae

options merger test

aff56a0

fix encryption mode comment

f25f21a

shreemaan-abhishek force-pushed the ai-proxy branch from 0af5364 to f25f21a Compare August 20, 2024 06:59

fix(lint): local ngx

d2d253e

shreemaan-abhishek requested a review from moonming August 20, 2024 08:22

shreemaan-abhishek added 3 commits August 21, 2024 09:35

fix lint

58ca8a7

index to json

f146f20

reindex

2317aa8

bzp2010 self-requested a review August 21, 2024 09:48

membphis reviewed Aug 22, 2024

View reviewed changes

apisix/plugins/ai-proxy.lua Outdated Show resolved Hide resolved

apisix/plugins/ai-proxy.lua Show resolved Hide resolved

apisix/plugins/ai-proxy/drivers/openai.lua Outdated Show resolved Hide resolved

apisix/plugins/ai-proxy.lua Outdated Show resolved Hide resolved

update doc

3429e03

kayx23 self-requested a review September 9, 2024 15:25

bzp2010 self-requested a review September 10, 2024 01:13

bzp2010 reviewed Sep 10, 2024

View reviewed changes

kayx23 reviewed Sep 10, 2024

View reviewed changes

shreemaan-abhishek mentioned this pull request Sep 10, 2024

docs: ai-* plugin docs should be moved into a new category and zh docs should be added #11563

Open

shreemaan-abhishek and others added 2 commits September 10, 2024 11:26

Apply suggestions from code review

7c08290

Co-authored-by: Traky Deng <[email protected]>

code review

977cf68

zhoujiexiong reviewed Sep 11, 2024

View reviewed changes

code review

073b3f8

zhoujiexiong reviewed Sep 11, 2024

View reviewed changes

shreemaan-abhishek added 5 commits September 11, 2024 10:57

code review

5831108

update priority

ab4c37e

update priority

cdd37db

fix test

e0e3e15

Merge branch 'master' of github.com:apache/apisix into ai-proxy

f6768e4

bzp2010 approved these changes Sep 12, 2024

View reviewed changes

moonming approved these changes Sep 13, 2024

View reviewed changes

nic-6443 approved these changes Sep 14, 2024

View reviewed changes

shreemaan-abhishek merged commit d46737f into apache:master Sep 17, 2024
33 of 34 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: ai-proxy plugin #11499

feat: ai-proxy plugin #11499

shreemaan-abhishek commented Aug 15, 2024 •

edited

Loading

bzp2010 left a comment •

edited

Loading

bzp2010 Sep 10, 2024

bzp2010 Sep 10, 2024

kayx23 Sep 10, 2024

shreemaan-abhishek Sep 10, 2024

kayx23 Sep 10, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

shreemaan-abhishek Sep 11, 2024

zhoujiexiong Sep 11, 2024

shreemaan-abhishek Sep 11, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

zhoujiexiong Sep 11, 2024

		@@ -334,6 +335,21 @@ function _M.get_body(max_size, ctx)
		end


		function _M.get_request_body_table()

	if core.table.try_read_attr(conf, "model", "options", "stream") then
	if request_table.stream then

	if conf.auth.query and type(conf.auth.query) == "table" then
	if type(conf.auth.query) == "table" then

	local path = (parsed_url.path or "/v1/chat/completions") .. query_params
	local path = (parsed_url.path or DEFAULT_URI) .. query_params

	return 500, "failed to send request to LLM server: " .. err
	return internal_server_error, "failed to send request to LLM server: " .. err

	// A simple loop that sends a message every 2 seconds
	// A simple loop that sends a message every 500ms

feat: ai-proxy plugin #11499

feat: ai-proxy plugin #11499

Conversation

shreemaan-abhishek commented Aug 15, 2024 • edited Loading

Description

Checklist

bzp2010 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shreemaan-abhishek commented Aug 15, 2024 •

edited

Loading

bzp2010 left a comment •

edited

Loading