Announcing AI Gateway

[subnetmarco]

Hello everyone 👋 , today we announced a set of new AI plugins in the broader Kong Gateway 3.6 release. You can read the full AI announcement here.

Essentially we can now use Kong Gateway to proxy AI traffic to cloud and self-hosted LLMs with a whole set of L7 AI Capabilities when it comes to prompts, observability and request/response transformation, and more. All other plugins can now be also used on top of upstream AI services to establish more advanced governance, security and traffic control patterns.

You can learn more about the new AI capabilities and even see demonstration videos at the official product page.

I am opening this discussion thread to collect feedback and ideas around this new AI use-case, and looking for feedback and suggestions from the community.

Update: Kong AI Gateway 3.8 has been announced with semantic caching, semantic guardrails, semantic routing and more.

[kandakji]

Hi Marco @subnetmarco , this is great. We'd be happy to contribute to this AI proxy to add support for Amazon Bedrock invoke model API in order to add it to the list of providers you support

[subnetmarco]

That would be awesome to have.

[tysoekong]

Hi @kandakji , I built the initial range of provider integrations to support this AI category.

Would really like to meet, so we can plan out adding support for the Bedrock Inference format! We want to enable as many providers and model variety as possible.

[subnetmarco]

Update: AWS Bedrock will be released in the upcoming v3.8 this summer, along with other LLM technologies.

[subnetmarco]

A new release of AI Gateway has been announced with streaming support, new plugins, richer analytics and more: https://konghq.com/blog/product-releases/ai-gateway-goes-ga

We are going to release AWS Bedrock and Gemini support in just a few weeks.

The new analytics allows to extract all sort of metrics from your GenAI usage:

[guleng]

@subnetmarco I have many LLM models for ai proxy, can I directly load balance and perform health checks on multiple models? I was unable to achieve load balancing for the LLM model in my actual Kong configuration

[guleng]

Do you have an approximate time?

[subnetmarco]

August/Sept.

[mepc36]

Hey @subnetmarco, is the August/September delivery date you mentioned here still accurate?

Thanks!

[subnetmarco]

Yes, second week of September.

[subnetmarco]

We just announced Kong AI Gateway 3.8 which includes more advanced load balancing algorithms, among many other semantic capabilities.

[abhijitherekar]

Hi @subnetmarco , I am from Acuvity AI , a genAI security startup and we would like to integrate with Kong AI gateway.

Please, let me know what is the best way to reach out to you.
Thanks

[Colstuwjx]

Hi team,

May I ask if we have any MCP Gateway feature updates or roadmap for this ? Just like higress ai gateway did, it supports to proxy existing open apis and act as a MCP Server Gateway in front of upstream apis, see blog post , any plan for Kong gateway to implement this kind of feature?

[subnetmarco]

Yes, we already support the ability to govern, secure and observe MCP traffic: https://konghq.com/blog/product-releases/securing-observing-governing-mcp-servers-with-ai-gateway

[Colstuwjx]

Hi @subnetmarco , thanks for your reply. However, from the blog post, I didn't see any features like configuring a http api in a MCP Server plugin like higress did, and also I've tried to search "MCP" keyword in plugin marketplace, still no record found: https://docs.konghq.com/hub/?search=mcp

May I know how to use this feature ? Or it's ONLY limited to a dedicated konnect MCP Server, which means we could ONLY use konnect MCP Server to govern kong gateway itself, rather than using kong gateway to proxy existing http apis (Kong as a rest-api-to-mcp-server proxy) or MCP Servers (Kong as a MCP Bridge)

Thanks.

[gperanich]

Hi @Colstuwjx, to expose a MCP server with Kong we would leverage the "rest-api-to-mcp-server proxy" as you mentioned above.

In blog (+underlying application), we are using native Kong Gateway routing to expose the MCP server. This grants the ability to layer policy on the MCP server API. For example, in the blog we use the following plugins to secure and govern the MCP server exposed via Kong Gateway:

• OpenID Connect - to secure the MCP server with AuthN + AuthZ
• HTTP Log - to integrate with our observability solution

Please have a look at the Kong Plugin Hub to see the other types of policy that can be leveraged to manage MCP servers with Kong.

[Colstuwjx]

we are using native Kong Gateway routing to expose the MCP server

But how could we expose those http services as MCP Server ? For the higress implementation, it supports to configure a http server endpoint in its MCP-Server wasm plugin (similar to Kong plugin), an example would be like this:

  matchRules:
  - config:
      server:
        name: random-user-server
      tools:
      - description: Get random user information
        name: get-user
        requestTemplate:
          method: GET
          url: https://fakerapi.it/api/v1/persons?_locale=en&_quantity=1
        responseTemplate:
          body: |-
            # User Information
            {{- with (index .data 0) }}
            - **Name**: {{.firstname}} {{.lastname}}
            - **Email**: {{.email}}
            - **Location**: {{.address.city}}, {{.address.country}}
            - **Phone**: {{.phone}}
            {{- end }}
    configDisable: false
    ingress:
    - user

By using this wasm plugin, higress could proxy the http api https://fakerapi.it/api/v1/persons and make it become a MCP Server. However, I didn't see any features like this in Kong, may I know how to achieve the same goal, for example, make a MCP Tool to invoke https://fakerapi.it/api/v1/persons api by using Kong as rest api to mcp server proxy ?

[subnetmarco]

To expose every API into an MCP server automatically will be an upcoming capability, it is not available yet.