AI Gateway

Why This Matters

Once a company has more than one team calling LLMs, the same problems appear on repeat: API keys scattered across services, no single place to see spend, no consistent way to fall back when OpenAI has an outage, and no shared cache so five teams pay five times for the same prompt. An AI gateway is the pattern that fixes this once instead of five times. It is the same lesson networking learned decades ago with API gateways for REST services, applied to a new kind of expensive, unreliable, fast-moving backend: the LLM.

Everyday Analogy

Think of a hotel's front desk. Guests do not call housekeeping, room service, and maintenance directly; they call the front desk, and the front desk routes the request, remembers the guest's preferences, keeps a log of every request, and calls a backup vendor if the usual supplier is out of towels. An AI gateway is the front desk for your LLM traffic: applications never talk to OpenAI or Anthropic directly, they talk to the gateway, which handles the routing, memory, logging, and backup plan.

What an AI Gateway Actually Does

A gateway is not a single feature; it is a bundle of infrastructure concerns that make sense to centralize once traffic crosses more than one team or more than one provider:

• Unified API surface. One request format (usually OpenAI-compatible) works against OpenAI, Anthropic, Google Gemini, Cohere, Mistral, or a self-hosted open-source model, so application code never rewrites its LLM-calling logic to switch providers.
• Auth and key management. Provider API keys live in the gateway, not scattered across a hundred microservices' environment variables. Applications authenticate to the gateway with their own credentials; the gateway holds the real provider keys.
• Routing and model selection. Requests can be routed by cost, latency, task type, or a fixed policy, applying the model-routing techniques from AI-039 (cheap model first, escalate to a frontier model only when needed) at the infrastructure layer instead of inside every application.
• Rate limiting and quota enforcement. Per-team, per-application, or per-user limits prevent one runaway job from exhausting a shared provider quota or blowing the monthly budget.
• Caching. Exact-match caching returns identical prior responses instantly; semantic caching (matching on embedding similarity, not exact text) catches paraphrased duplicate requests too, cutting both latency and provider cost.
• Logging and observability hooks. Every request and response is logged centrally, feeding the dashboards, traces, and cost breakdowns covered in AI-028, without each application team building its own logging pipeline.
• Fallback and retry across providers. If Anthropic returns a 529 (overloaded) or times out, the gateway can retry against a fallback provider automatically, so an outage at one vendor does not become an outage in your product.
• Guardrail and policy enforcement. Content filtering, PII redaction, and prompt-injection checks (see AI-023) can run at the gateway, giving one enforcement point instead of relying on every team to remember to add them.

Real Gateway Products

Most teams start with LiteLLM (free, self-hosted, fast to stand up) and graduate to a managed product like Portkey or a cloud-native gateway once they need enterprise SSO, SLAs, or compliance certifications.

A worked example: a request's journey through a gateway

Say a customer-support app sends "How do I reset my password?" through the gateway. The sequence looks like this:

1. Auth check. The gateway validates the app's API key and checks it against the support team's rate limit and monthly budget.
2. Semantic cache lookup. The gateway embeds the incoming prompt and compares it against cached embeddings. If a semantically similar question ("I forgot my password, help") was answered in the last hour, the cached answer returns immediately, no provider call, no cost, sub-50ms latency.
3. Cache miss → routing. No match found. The gateway's routing policy says: try a fast, cheap model (say, a smaller open model) first for simple support queries, using classification logic like AI-039's model-routing pattern.
4. Provider call and fallback. The gateway calls the chosen provider. If that provider times out or returns a rate-limit error, the gateway automatically retries against a configured fallback provider rather than surfacing an error to the user.
5. Guardrail pass. The response is checked for PII leakage and policy compliance before being returned.
6. Logging. The full round trip (prompt, model used, latency, cost, cache hit/miss, fallback triggered or not) is written to the observability pipeline for the dashboards in AI-028.

The application code that sent the original request never knew which provider actually answered, whether it came from cache, or whether a fallback fired. That invisibility is the entire point.

Real-World Showcase

• Companies running multi-model products (chatbots that route between GPT-4o-class and Claude-class models based on task) use gateways like Portkey or LiteLLM specifically to avoid hard-coding provider logic into every service.
• Cloudflare reports that AI Gateway customers commonly see double-digit percentage cost reductions purely from caching repeat prompts, without changing a single line of application code.
• Enterprises with existing Kong API gateway deployments extend the same infrastructure (and the same platform team's expertise) to LLM traffic via Kong AI Gateway rather than standing up a separate system.

Try It Yourself

1. Sketch a simple architecture diagram: three application services, two LLM providers, and a gateway in between. Label which concerns (auth, cache, routing, logging) live in the gateway versus in each application.
2. Pick two gateway products from the table above and list one scenario where you would choose each over the other (for example: "already all-in on Cloudflare" vs. "need full control over open-source deployment").
3. Design a fallback policy in plain English for a support chatbot: which provider is primary, which is the fallback, and what triggers the switch (timeout threshold, error codes, or both)?

Common Mistakes

• Skipping a gateway "because we only use one provider today" and then rewriting every service's LLM-calling code when a second provider is added later.
• Caching prompts that contain user-specific PII without partitioning the cache per user or tenant, leaking one user's data into another user's cached response.
• Treating the gateway as only a cost-saving tool and ignoring its role in reliability (fallback) and governance (guardrails, audit logs).
• Routing purely on cost without checking output quality, sending tasks that need a frontier model to a cheaper model that cannot actually do them.
• Storing provider API keys in application config instead of centralizing them in the gateway, defeating half the point of adopting one.

Key Takeaways

• An AI gateway centralizes auth, routing, caching, rate limiting, logging, and fallback for all LLM traffic behind one API.
• Real products span self-hosted (LiteLLM, Kong AI Gateway) to managed (Portkey, Cloudflare AI Gateway, AWS Bedrock Gateway, Azure API Management for AI).
• Semantic caching and cross-provider fallback are the two features that most directly cut cost and improve reliability.
• Gateways make the model-routing cost techniques from AI-039 and the observability practices from AI-028 enforceable infrastructure-wide, not per-team habits.
• Adopt a gateway before you have a second provider or a second team, not after the rewrite becomes painful.