Announcing AI Gateway

[subnetmarco]

Hello everyone 👋 , today we announced a set of new AI plugins in the broader Kong Gateway 3.6 release. You can read the full AI announcement here.

Essentially we can now use Kong Gateway to proxy AI traffic to cloud and self-hosted LLMs with a whole set of L7 AI Capabilities when it comes to prompts, observability and request/response transformation, and more. All other plugins can now be also used on top of upstream AI services to establish more advanced governance, security and traffic control patterns.

You can learn more about the new AI capabilities and even see demonstration videos at the official product page.

I am opening this discussion thread to collect feedback and ideas around this new AI use-case, and looking for feedback and suggestions from the community.

Update: Kong AI Gateway 3.8 has been announced with semantic caching, semantic guardrails, semantic routing and more.

[kandakji]

Hi Marco @subnetmarco , this is great. We'd be happy to contribute to this AI proxy to add support for Amazon Bedrock invoke model API in order to add it to the list of providers you support

[subnetmarco]

That would be awesome to have.

[tysoekong]

Hi @kandakji , I built the initial range of provider integrations to support this AI category.

Would really like to meet, so we can plan out adding support for the Bedrock Inference format! We want to enable as many providers and model variety as possible.

[subnetmarco]

Update: AWS Bedrock will be released in the upcoming v3.8 this summer, along with other LLM technologies.

[subnetmarco]

A new release of AI Gateway has been announced with streaming support, new plugins, richer analytics and more: https://konghq.com/blog/product-releases/ai-gateway-goes-ga

We are going to release AWS Bedrock and Gemini support in just a few weeks.

The new analytics allows to extract all sort of metrics from your GenAI usage:

[guleng]

@subnetmarco I have many LLM models for ai proxy, can I directly load balance and perform health checks on multiple models? I was unable to achieve load balancing for the LLM model in my actual Kong configuration

[guleng]

Do you have an approximate time?

[subnetmarco]

August/Sept.

[mepc36]

Hey @subnetmarco, is the August/September delivery date you mentioned here still accurate?

Thanks!

[subnetmarco]

Yes, second week of September.

[subnetmarco]

We just announced Kong AI Gateway 3.8 which includes more advanced load balancing algorithms, among many other semantic capabilities.