This guide is for the architect or platform lead who has been asked whether the enterprise needs an AI gateway, and for the compliance or risk leader who will be asked to sign off on one. It sets out what an AI gateway does that an API gateway does not, where it sits in a regulated architecture, which products occupy the space and how they differ, what fails in production, and when the honest answer is that you do not need one yet. It is written for people who already run an API gateway and already call language models, and who now have to decide whether those two facts require a third thing.
An AI gateway is a policy enforcement point between applications and the language models they call. It authenticates the caller, applies limits on tokens and cost, inspects prompts and responses, routes requests to the right model, and records what was sent and returned. It does for model traffic what an API gateway does for API traffic, with controls that only make sense when the payload is a prompt and the bill is measured in tokens. The rest of this guide is about the decisions behind that sentence.
Eight Things That Change
The first question a CIO asks is reasonable: the enterprise already owns an API gateway, so why is this not another route on it? The answer is that eight things change when the thing being called is a model rather than a service. A product that does none of them is an API gateway with a new label.
Token accounting and budgets come first. An API call costs roughly the same whether the payload is ten bytes or ten kilobytes. A model call is billed by the token, in and out, and a long document or a verbose agent can cost a hundred times more than a short question. The gateway counts tokens per request, attributes them to an application, a team or a project, and enforces a budget before the model is called rather than after the invoice arrives. Prompt and response inspection is second. The gateway reads what goes in and what comes out. On the way in it can block or flag a prompt that matches a policy, such as an attempt to place a customer record into a prompt that should not contain one. On the way out it checks the response before the caller sees it.
Model routing and fallback is third. A request that needs a small, fast model should not go to a large, expensive one, and a request that carries regulated data should not leave the building. The gateway sends each request to the model that fits its cost, latency and data-handling requirement, and fails over when a provider is down or rate-limited. Semantic caching is fourth: two prompts close enough to mean the same thing can share an answer, which cuts cost and latency on repeated questions and introduces a class of bug covered later. Redaction on the way out is fifth. When the model is not inside your boundary, the gateway strips or masks personal and regulated data before the prompt leaves.
Tool-call authorization is sixth, and it is the one most vendor demonstrations skip. Agents do not only ask questions. They call tools: search a system, open a ticket, move money. The gateway decides which agent may invoke which tool, under whose authority, before the call is made. This is the control that separates agent traffic from chat traffic. Per-model rate limits are seventh, by model, by caller and by time window, so one runaway agent in a loop cannot exhaust a quota shared by forty applications. Audit of prompts and completions is eighth. The gateway records the prompt, the response, the model and version, the caller, and every policy decision, in a form a model-risk reviewer can read months later. Not a log line. A record.
The Response Path and Architecture Placement
The gateway sees every request twice, and the second pass is the one that gets dropped. On the way to the model it authenticates the caller, checks the budget, inspects the prompt, picks the model and forwards the call. On the way back it inspects the response, redacts what must not leave, records the exchange and returns the answer. A gateway that only governs the outbound half will return a response containing an account number the model reconstructed from context, and log nothing about it. The first thing to check in any gateway design is whether the response path has a policy on it at all. Draw the five stops on a whiteboard, application, gateway, model, gateway, application, and mark which ones run inside your boundary. That drawing is the design.
Where the AI gateway sits relative to the API gateway is a secondary question with three defensible answers. It can sit in front, so model traffic is governed before it reaches the estate, which fits when model traffic comes from many sources and the API gateway is not the natural chokepoint. It can sit behind, so the API gateway handles identity and coarse routing and hands model-bound traffic to an AI gateway for model policy, which fits when the API gateway is already the front door for everything. Or it can be the same product, with the API gateway vendor's AI policies applied as a second policy set on model routes. The third arrangement is the most common in practice, because the enterprise already owns an API gateway and every major vendor has shipped AI capabilities into it. The arrangement matters less than one question. Your model traffic is only as governed as the component that sees the prompt in the clear. Identify that component and where it runs, and the rest of the architecture follows.
Regulated Requirements and Policy Ownership
Vendor pages describe the gateway as a way to call several hosted providers under one key, with a dashboard. That is a real use, and it is not the use a bank or a hospital has. In a regulated enterprise the gateway is the control that lets you say, in writing, what a model was allowed to see, what it was allowed to do, and who authorized it. Three requirements change the design. The prompt must not leave the boundary unless a policy says it may, which means either the gateway runs inside the boundary and routes to in-perimeter models, or it redacts before egress under a written policy that someone has reviewed. The audit record must be complete enough for a model-risk reviewer to reconstruct a decision months later: prompt, response, model and version, caller, and every policy applied, retained as long as the decision it supported. A dashboard of request counts does not meet this; a queryable record does. And tool-call authorization must be enforced by identity. An agent that calls a core banking system does so as a named principal with scoped permissions, not through a shared key the gateway happens to hold. The gateway enforces the policy; the identity layer defines who the agent is and what it may touch, a design covered in AI agent governance on WSO2. Where the tools behind the gateway speak the Model Context Protocol, the injection surface moves into tool results, which is the the subject of an earlier piece on MCP and authorization.
There is a fourth requirement that is not technical. Someone has to own the policies. Gateway deployments stall far more often on unassigned routing rules, budgets and redaction patterns than on the software. Budget for that role before the licence.
Vendor Landscape and Common Failures
The landscape is a map, not a ranking, and it changes quarterly, so verify any capability with the vendor before a design depends on it. WSO2's AI Gateway covers token and cost tracking, guardrails including PII masking, semantic caching, multi-provider proxying and MCP tool exposure, and ships both as the Bijira SaaS and as self-hosted software (the deployment trade-off between the two is is set out in Bijira vs API Manager). Kong's AI plugins add add token-based rate limiting, semantic caching and load balancing across models to the Kong gateway, and are strongest where Kong already fronts the APIs. Apigee's AI gateway capabilities bring model routing, token quotas and Model Armor prompt and response sanitization to Google Cloud API management, and fit naturally in a Google Cloud estate. Cloudflare AI Gateway is edge-hosted caching, logging and rate limiting for calls to hosted models. Portkey is a dedicated LLM gateway focused on routing, observability and cost across providers. LiteLLM is an open-source proxy that presents many providers behind one interface, light and self-hostable, and a common first step. AWS, Azure and Google each offer model-traffic controls that work best inside their own platform and least well across platforms. If your compliance regime requires the control plane inside your boundary, the list shortens to the self-hostable options. If it can accept a hosted control plane with in-perimeter traffic, the list is longer and the decision moves to operational fit.
Four failures account for most gateway incidents, and none shows up in a demonstration. Streaming and timeouts: model responses stream for seconds or minutes, and a gateway tuned for millisecond API calls will cut them off, buffer them so the client sees nothing until the end, or time out the connection. Every timeout between application and model has to be raised deliberately, and streaming has to be tested through the gateway, not around it. Token accounting drift: the gateway counts tokens one way and the provider bills another, because tokenizers differ by model and cached or truncated responses are counted inconsistently. Budgets that look enforced are not, until the invoice shows the gap. Reconcile the gateway's count against the provider's bill monthly and treat a persistent gap as a defect. Cache invalidation: a semantic cache keyed on the user prompt keeps serving old answers after the system prompt or the model changes, so the cache key has to include the system prompt, the model and the version, or the cache has to be flushed on every change to any of them. Most teams learn this the day a policy change appears not to take effect. Retries multiplying spend: the client retries, the gateway retries, the provider SDK retries, and one failed call becomes nine billed calls while a provider outage becomes a spend spike. Retries belong in one place, and the gateway is that place, with the client and SDK retries turned off.
A proxy is a weekend and a gateway is a program. Routing and logging can be built in a few hundred lines, and many teams start there. The problem is what comes next. Token budgets that reconcile to the invoice, semantic caching that invalidates correctly, tool-call authorization tied to an identity provider, and an audit store a reviewer will accept are each a project of their own, and a team that builds them all ends up owning a product. If the enterprise already runs an API gateway with AI policies, extend it. If not, adopt an open-source gateway or buy one, and spend the engineering on the policies, because the policies are the part no vendor can write. They encode what your enterprise is willing to let a model see and do.
The half nobody writes is when you do not need one. One application, one hosted model, one team, no regulated data in the prompt, and spend small enough that the provider's own dashboard is enough control. That describes most first projects, and a gateway would add latency and an operations burden for nothing. Add it when any of these arrives: the second team, the second provider, the first in-perimeter model, the first agent that calls a tool, or the first question from an auditor. At that point it stops being optional, and it is far easier to add before the fifth application than after. The migration cost is not the gateway. It is moving five applications' worth of keys, retries and error handling behind it.
An AI gateway is worth having when model traffic has more than one source, more than one destination, or more than one person who will be asked about it later. In a regulated enterprise that is nearly always, and the deciding question is not which product but which component sees the prompt in the clear and where it runs. Answer that, assign an owner to the policies, test the response path as carefully as the request path, and the product choice becomes the easy part.
