What is an AI Gateway — Who are the leading open-source projects and de facto standards?

Even if an AI application initially calls only one OpenAI API, the situation changes once it enters the operational phase. Each department may use Anthropic, Gemini, AWS Bedrock, Azure OpenAI, and in-house GPU models together, need to switch to alternative models in case of provider failure, and manage per-user costs and data export policies.

If each application implements these features directly, authentication keys and retry code will be scattered across services, and request formats and error handling will need to be rewritten every time a model is changed. An AI Gateway is an intermediary layer positioned between applications and multiple model providers, consolidating model calls into a single point of policy and observability.

This article explores how an AI Gateway differs from a general API Gateway, what protocols and data actually flow internally, and which to choose among LiteLLM, Envoy AI Gateway, Kong, Apache APISIX, Higress, and Portkey. Finally, it clearly answers the question: “Who is the de facto standard for AI Gateways?”

To start with the conclusion: No single product is the de facto standard

As of August 2026, there is no single official standard or product dominating the entire AI Gateway market. Instead, the answer becomes clear when we divide it into layers.


Question	Closest Answer Currently	Points to Note

—	—	—
What call format do applications use?	OpenAI Compatible HTTP/JSON API	Not an IETF or CNCF standard, but a widely adopted de facto interface convention.
Who is the representative of general-purpose multi-provider OSS Gateways?	LiteLLM Proxy	Broad feature set and ecosystem, but need to check the boundary between OSS and commercial features.
What is the standard for self-hosted inference routing in Kubernetes?	Gateway API Inference Extension	Focuses on optimizing in-cluster GPU inference rather than integrating external SaaS models.
What about Kubernetes·Envoy-based AI Gateway implementations?	Envoy AI Gateway	A rapidly maturing project, but not the sole winner in the overall general-purpose market.
What are the extension options for existing enterprise API management?	Kong, Apache APISIX, Higress	Depends on the Gateway already used by the organization and the required AI plugins.

Therefore, to summarize in one sentence:

The de facto interface is the OpenAI compatible API, and the project closest to the de facto general-purpose open-source implementation is LiteLLM. However, in Kubernetes internal inference routing, Gateway API Inference Extension and Envoy-based solutions create a separate standard axis.

GitHub Stars do not prove installation numbers or quality, but they serve as a supplementary indicator of ecosystem interest. The official LiteLLM repository currently positions itself as a Gateway that calls over 100 LLM APIs in OpenAI or native formats, boasting over 50,000 Stars and active releases. Due to this versatility, it is one of the first OSS projects considered when building a new AI platform for multi-model entry points. However, this assessment is a practical judgment based on adoption and compatibility, not an indication of an “official standard.”

What Exactly Does an AI Gateway Do?

General API Gateways also perform TLS termination, authentication, rate limiting, routing, and logging. An AI Gateway adds the ability to understand the semantics of model calls.

model, messages, input, tools, stream 같은 LLM 요청 필드 해석
Conversion of OpenAI formats to provider-specific formats like Anthropic Messages, Gemini, Bedrock, etc.
Routing based on model name, cost, latency, failure status, and data classification
Token-level usage, per-model pricing, and per-user/project budget aggregation
Handling long streaming connections and provider-specific errors/retries
Integration with PII, prohibited content, and Prompt Injection checks for Prompts and Responses
Caching of identical/similar Prompts and Semantic Cache
Application of policies based on model features such as Tool Call, Embedding, image, and speech

Figure: Typical Request/Response Path of an AI Gateway · Reconstructed based on official documentation from LiteLLM, Kong AI Gateway, and Envoy AI Gateway

When a request comes in, the Gateway first verifies the user’s Virtual Key or JWT and checks the tenant, budget, and model allowlist. It then converts the common request into the target provider’s format and authentication method, and forwards it to the appropriate model or deployment. In the response, it records token usage and latency, applies output guardrails if necessary, and then returns the stream to the client.

The core idea is not merely “adding another proxy.” It’s about elevating scattered model-specific access code and operational policies from applications to a central Model Access Layer.

Does it use HTTP, and what is the internal data format?

Most AI Gateway external interfaces use REST-style APIs over HTTPS with JSON in the body. Streaming output commonly uses Server-Sent Events (SSE). The OpenAI Responses API also returns SSE events when stream=true.

A typical request structure is as follows:

POST /v1/chat/completions HTTP/1.1
Authorization: Bearer virtual-key
Content-Type: application/json

{
  "model": "customer-support-model",
  "messages": [
    {"role": "user", "content": "환불 정책을 알려줘"}
  ],
  "stream": true
}

Here, customer-support-model might not be the actual provider’s model name. It can be treated as a logical model name within the Gateway and linked to multiple deployments as follows:

customer-support-model
├─ 70% → Azure OpenAI 배포 A
├─ 20% → AWS Bedrock 모델 B
└─ 장애 시 → 사내 vLLM 모델 C

When the Gateway communicates with the provider, even if it’s still HTTP/JSON, the URL, Header, and Body Schema will differ. AWS Bedrock may require SigV4 authentication, Anthropic uses fields and headers specific to its Messages API, and Gemini uses its own generateContent structure. The Gateway translates these differences.

Is OpenAI Compatibility a Standard Specification?

No. While structures like POST /v1/chat/completions, messages, tools, and stream have been widely implemented, becoming a de facto common language, they are not independent standards managed by IETF RFC, W3C, or CNCF. They are compatibility implementations of OpenAI’s product API contract by other projects.

Furthermore, even if something is labeled “OpenAI compatible,” the scope of compatibility varies.

Only Chat Completions might be supported, not Responses API
Parallel execution of Tool Calls or Structured Output might differ
Support for Image, Speech, Batch, and Embedding Endpoints may vary
SSE Event names and termination methods might differ
Prompt Cache, Reasoning Token, and Usage fields might be missing
Provider-specific features might not be expressible in a common Schema

Therefore, compatibility should be tested as a combination of Endpoint × Model × Feature × Streaming × Error Handling, not just a single checkbox.

Figure: AI Gateway Protocol Layers and De Facto Interfaces · Reconstructed based on documentation from OpenAI API, LiteLLM, and Higress AI Proxy

When is an AI Gateway Needed?

For a small service calling a single model, a Gateway might actually increase operational overhead. Its value increases when several of the following conditions arise:

Multiple Models and Providers

If you need to switch or use models in parallel due to price, performance, data locality, or failure risk, a common Endpoint and logical model name are useful. You can change the Gateway’s routing policy without modifying the application.

Need for Organization-Wide Authentication and Cost Control

Instead of distributing provider API Keys to all teams, issue short-lived tokens or Virtual Keys from the Gateway. This allows central enforcement of model allowlists, monthly budgets, and TPM/RPM limits per user, department, or project.

Failure Response and Retries are Critical

LLM calls take longer than general APIs, and rate limits or temporary capacity shortages occur frequently. A Gateway consistently applies timeouts, exponential backoff, circuit breakers, and fallbacks. However, if a streaming request that has already started responding is switched to another model, issues like duplicate sentences or billing discrepancies may arise, so the retryable points must be clearly defined.

Security and Auditing are Required

It’s necessary to track who sent what data classification to which model and to prevent sending Prompts containing personal information to external models. The Gateway is an ideal point to integrate DLP, Guardrails, and audit logs.

However, if raw Prompts and Responses are unconditionally stored in the Gateway, it becomes the largest repository of personal information. It is safer to default to not storing the body, and if necessary, to separate masking, encryption, retention periods, and access rights.

Comparison of Leading Open-Source AI Gateways


Project	Best Fit Environment	Strengths	Points to Check

—	—	—	—
LiteLLM Proxy	AI platform teams rapidly building general-purpose multi-model Gateways	100+ providers, OpenAI compatible Endpoint, Virtual Key, cost/budget, Routing/Fallback, broad integrations	Python-centric operations, frequent release upgrade validation, distinction between OSS and commercial features
Envoy AI Gateway	Organizations using Kubernetes·Gateway API·Envoy as their standard data plane	Envoy Gateway-based, CRD, provider conversion, hierarchical central/team Gateways	Relatively new project, closer to declarative infrastructure than operational UI
Kong AI Gateway	Enterprises extending existing Kong API management to AI calls	Combines existing authentication·Plugins·Analytics with AI Proxy, Provider Normalization	Check licenses and editions for advanced features like Semantic Routing·Cache
Apache APISIX	Organizations aiming to manage high-performance API Gateway and AI traffic on a single platform	Apache 2.0, LLM Proxy·Load Balancing·Retry·Token Rate Limit, MCP Bridge	More of a plugin combination nature than a dedicated AI management experience
Higress	Environments managing Envoy·Istio·Wasm, Chinese and global models, and MCP together	OpenAI contract-based AI Proxy, multi-model·Cache·Token Limit, MCP Hosting	Difference between open-source and Alibaba Cloud commercial versions, suitability for operational ecosystem
Portkey Gateway	Teams requiring lightweight Edge·Node environments and code-based Routing·Guardrail configuration	MIT, fast Proxy, Retry·Fallback·Load Balance·Guardrail	2.0 is Pre-release, need to distinguish Hosted·Enterprise features

LiteLLM Proxy — The First Candidate for General-Purpose OSS

LiteLLM provides both a Python SDK and a central Proxy Server. Applications only need to change the base_url of the OpenAI SDK to LiteLLM, and LiteLLM handles the conversion of provider-specific requests, responses, and errors.

Its strength lies in its scope of support. It handles various Endpoints such as Chat, Responses, Embedding, image, speech, and Batch, as well as multiple providers within a single project. Features like Virtual Key, per-user/team costs, Budget, Load Balancing, Fallback, and Observability also align with the needs of AI platform teams.

Therefore, if your goal is to “initially bundle multiple external and in-house models into a single API,” it’s worth conducting a PoC with LiteLLM first. However, as features are rapidly added, it’s essential to fix stable images, review change logs, and test for load, streaming, and database failures.

Envoy AI Gateway — A Choice Closer to the Kubernetes Data Plane

Envoy AI Gateway manages access to generative AI services on top of Envoy Gateway. It declares Providers, Routes, and Backends using Kubernetes Gateway API and CRDs, and performs request transformation, authentication, and traffic policies within the Envoy data plane.

If your platform team already operates Envoy, Gateway API, GitOps, and OpenTelemetry, adopting Envoy AI Gateway is more natural than introducing a separate Python Gateway. It also fits well with a 2-tier architecture where a central Gateway handles authentication and global rate limits, and team or model-tier Gateways manage detailed routing.

Meanwhile, Kubernetes’ Gateway API Inference Extension addresses a slightly different problem. Rather than translating external SaaS Providers into a single API, it focuses on selecting the optimal Endpoint among self-hosted Model Server Replicas (like vLLM), considering KV Cache, Queue, LoRA, and accelerator status. Official documentation also explains that higher-level AI Gateways like LiteLLM can be combined with Kubernetes Inference Gateways.

Kong AI Gateway — When Reusing Existing API Management Assets

Kong converts OpenAI-formatted requests to various Providers via its AI Proxy Plugin and integrates with Kong’s existing authentication, rate limiting, and logging. If Kong Konnect or Kong Gateway is already the enterprise standard, the significant advantage is the ability to leverage existing operations teams, policies, and observability systems.

However, stating that “all Kong AI Gateway features are open source” is inaccurate. Features usable as an AI Proxy should be distinguished from advanced governance features like Semantic Cache, which require an Enterprise License, before adoption.

Apache APISIX — When Adding AI Capabilities to a General-Purpose API Gateway

Apache APISIX is an Apache 2.0 API Gateway offering dynamic Routing, authentication, Circuit Breaker, and Observability. With plugins like ai-proxy, ai-proxy-multi, and Token Rate Limit, it can configure LLM request transformation, load balancing, and fallback among multiple models.

It is suitable when you want to operate existing HTTP·gRPC·microservice traffic and AI APIs on a single high-performance Gateway. It also provides features like MCP Bridge, which exposes existing stdio MCP Servers as HTTP SSE services, allowing for consideration of a common entry point for model and tool APIs.

Higress — When Integrating AI, API, and MCP with Envoy·Wasm

Higress is an AI Native API Gateway based on Istio and Envoy, extensible with Go·Rust·JavaScript Wasm Plugins. Its AI Proxy is based on the OpenAI API contract and handles multi-model Proxy, Token Rate Limit, Cache, WAF, authentication, and MCP Server Hosting within a single project.

Having evolved within Alibaba’s internal and cloud environments, it is attractive to organizations using that ecosystem and multiple Chinese models. Conversely, for domestic organizations, it is advisable to verify documentation, support, operational experience, and the functional boundaries between the Open Source and Commercial versions during a PoC.

Portkey Gateway — Lightweight Gateway and Configuration-Based Routing

Portkey Gateway is a lightweight MIT project based on TypeScript. It allows defining Retry, Fallback, Weighted Load Balancing, and Guardrails with JSON Config and can be called by OpenAI-compatible clients. It becomes a comparison candidate when a lightweight, Node·Edge-friendly proxy is desired.

The official repository currently announces 2.0 as a Pre-release, where the Enterprise Gateway Core is merged into OSS. Since there are also Hosted·Enterprise-only features marked with an asterisk in the feature table, it’s necessary to distinguish between the actual deployment branch and the licensed/operational features.

Which Project Should You Choose?

Figure: AI Gateway Selection Guide by Operating Environment · Reconstructed based on official documentation of each project

The selection criteria should be based on the organization’s existing operational practices rather than the number of features.

If Rapid Multi-Model Integration is a Priority, Choose LiteLLM

If you are a Python-based AI platform team, OpenAI SDK compatibility is important, and you want to quickly bundle external Providers with internal models like vLLM·Ollama, then validating LiteLLM first is a reasonable approach.

If Kubernetes and Declarative Network Policies are a Priority, Choose Envoy

If Gateway API, GitOps, Envoy, and OpenTelemetry are already standard, consider Envoy AI Gateway. If optimizing self-hosted GPU Model Serving is key, you should also look at the Gateway API Inference Extension and Endpoint Picker layers like llm-d Router.

If You Have an Existing API Gateway, First Look at Extension Possibilities

If you are already stably operating Kong, APISIX, or Higress, check if the AI Plugin of your existing product meets your requirements before adding another new Gateway. The cost of creating duplicate authentication, auditing, and disaster recovery systems can be surprisingly high.

If a Very Lightweight Edge Proxy is Needed, Choose Portkey

If you need to start quickly in a Node environment and code-centric Config, Retry·Fallback, and basic Guardrails are important, Portkey might be suitable. However, you need to verify if you can accept the boundaries between pre-release and commercial features.

Checklist for Mandatory Validation in PoC

Choosing based solely on the product’s supported features list in its introduction can lead to problems in the operational phase. At a minimum, the following items must be tested with actual traffic:

Feature Compatibility: Do Chat, Responses, Tool Call, Structured Output, Embedding, Image, and Streaming actually work with the target model?
Failure Semantics: In case of 429, 5xx, Timeout, or Stream interruption, which requests are retried, and are there duplicate charges or duplicate outputs?
Authentication and Authorization: Is the Provider Key not exposed to the application, and are user·project·model-level permissions enforced?
Cost Accuracy: Are Cached Tokens, Reasoning Tokens, image/speech costs, and Provider discounts calculated correctly?
Data Protection: Are Prompt·Response·Headers not left in raw form in logs, traces, errors, and admin screens?
Performance: Are the additional latency for short requests, the number of long SSE connections, backpressure, and the impact of Gateway failures within acceptable limits?
Operational Recovery: Does the existing Data Plane operate safely even if the configuration DB or Control Plane stops?
Upgrades: Can API Schema, DB Migration, Plugin, and Model Provider changes be validated with Canary deployments?
Licensing: Are the boundaries between OSS Core, Commercial Plugin, and Hosted Service, along with support conditions, documented?

In particular, simply stating “fallback is supported” is insufficient. Falling back to a different region of the same model family versus falling back to an entirely different model can result in variations in answer quality, Tool Schema, and safety policies. It is necessary to compare the normal response rate and semantic changes using a task-specific Golden Dataset.

Recommended Reference Architecture

As scale increases, it’s better to divide functionalities into layers rather than consolidating everything into a single Gateway.

애플리케이션
→ WAF / 일반 API Gateway
→ AI Gateway
   ├─ 사용자·테넌트 인증
   ├─ 모델 허용 목록·예산·Rate Limit
   ├─ PII / Guardrail 연계
   ├─ 논리 모델 Routing·Fallback
   └─ 토큰·비용·품질 관측
→ 외부 Model API
또는
→ Kubernetes Inference Gateway
→ vLLM / TGI / 사내 Model Server

A general API Gateway handles internet boundaries and organization-wide API policies, while an AI Gateway handles policies that understand model semantics. Within a self-hosted GPU Cluster, an Inference Gateway selects replicas considering Queue and KV Cache states.

This architecture also makes product replacement easier. The higher-level AI Gateway can maintain OpenAI-compatible Endpoints, while lower-level Inference Gateways or Providers can be changed incrementally.

Conclusion

An AI Gateway is not just a simple Reverse Proxy. It is the entry point for an AI platform that transforms HTTP/JSON APIs from multiple model providers into a common interface, and manages authentication keys, costs, routing, disaster recovery, observability, and security policies from a single location.

Currently, there is no “AI Gateway standard product” defined by ISO or IETF in the market. Instead, the OpenAI compatible API has become the de facto common language at the application layer, and LiteLLM is the closest to being the representative choice for general-purpose multi-provider OSS. In Kubernetes self-hosted inference, the Gateway API Inference Extension and Envoy-based solutions are creating a separate, standardized axis.

Therefore, the answer depends on the operating environment, not a single logo.

If you want to quickly bundle multiple models, LiteLLM is the choice. If you want to standardize Kubernetes networking and GPU inference, Envoy·Gateway API family is suitable. If you already have an API Gateway, considering extending Kong·APISIX·Higress first is a realistic starting point.