---
title: "How to choose an AI gateway for agents"
description: "Choose an AI gateway by the requests and policies it must preserve. Model count and a successful chat demo are weak substitutes for testing your tools, streams, credentials, fallbacks, and usage recor"
canonical: https://caveman.so/guides/choose-ai-gateway
last-updated: 2026-09-07
---

# How to choose an AI gateway for agents

Choose an AI gateway by the requests and policies it must preserve. Model count and a successful chat demo are weak substitutes for testing your tools, streams, credentials, fallbacks, and usage records.

If the immediate problem is repeated agent context, a gateway replacement may be unnecessary. You can evaluate [Caveman compression](/guides/prompt-compression) as a separate change while keeping the gateway that already works.

## Write down your request contract

Inventory actual APIs and features used by the application: Chat Completions, Responses, Anthropic Messages, Gemini requests, tool calls, structured output, images, streaming, and provider-specific state.

“OpenAI compatible” is a starting point, not a complete contract. Test the fields and event types your client sends and consumes. A gateway can support ordinary text while handling a less common feature differently.

Keep provider model IDs distinct from gateway aliases. Record how the caller selects a model and which credential authorizes it.

## Decide where keys and policy live

| Requirement | Evidence to ask for |
| --- | --- |
| Provider-key custody | Where secrets are stored and which process uses them |
| Virtual keys | Per-caller restrictions tested with allowed and forbidden requests |
| Budgets and rate limits | Behavior at the limit, including concurrent calls and retries |
| Provider or region pins | The route remains constrained during fallback |
| Guardrails | Which input and output paths they inspect, including streams |
| Auditability | A request can be traced to the caller and policy decision |

Test refusal behavior as carefully as the happy path. A candidate that forwards a forbidden model request has failed even if the answer is correct.

A local Caveman proxy is not a substitute for all gateway governance. It listens on loopback and serves supported local request paths. Caveman's hosted Router and Platform require private access.

## Identify retry and fallback ownership

An SDK may retry, the gateway may retry, and a routing policy may escalate. Decide which layer owns each transition and what evidence records the extra provider call.

Use a staging provider failure or deterministic fixture to inspect behavior. Check whether tools can execute twice, whether partial streams are retried, and whether the fallback preserves required capabilities.

Keep timeouts meaningful across the whole task. A series of individually reasonable retry delays can exceed the user's total wait budget.

## Test each cache separately

Provider prompt caching, gateway response caching, and application result caching have different semantics. Ask which one the gateway controls and how a hit appears in logs and billing evidence.

Response reuse needs correct freshness and access boundaries. Provider prefix reuse needs the right model and stable context. Do not merge both into a single unexplained savings number.

Run cold and warm tests using [the caching method](/guides/prompt-caching). Include writes, misses, and a request that should never reuse another caller's result.

## Evaluate observability as a contract

Inspect raw usage, model identity, request status, latency, retry relationships, and task attribution. Check how missing counters and unpriced models appear.

Confirm which data is captured, where redaction occurs, and how retention works. Content-based investigation requires different evidence from usage-only reporting. Keep the distinction explicit.

Test duplicate instrumentation and exporter failure. The candidate should make loss visible and preserve the application's intended reliability behavior. [Agent observability](/guides/agent-observability) gives a field-level starting point.

## Compare candidates by deployment fit

| Candidate | Comparison to start with |
| --- | --- |
| LiteLLM | [Gateway controls and adding Caveman](/compare/litellm) |
| OpenRouter | [Provider access, routing, and context cost](/compare/openrouter) |
| Portkey | [Gateway governance and optimization](/compare/portkey) |
| Helicone | [Gateway observations and full-task cost](/compare/helicone) |
| Cloudflare AI Gateway | [Gateway controls and local compression](/compare/cloudflare-ai-gateway) |
| Vercel AI Gateway | [Hosted model access and agent context](/compare/vercel-ai-gateway) |
| Bifrost | [Gateway performance and task economics](/compare/bifrost) |

Review the current edition, deployment model, and commercial terms for each candidate. A feature in an enterprise product page may not exist in the self-hosted package you plan to install.

## Measure the cost of operating the gateway

Include provider charges, gateway fees, infrastructure, observability storage, and the engineering work required to maintain the deployment. Keep measured usage separate from catalog estimates and negotiated bills.

Latency benchmarks need representative concurrency, payload size, streaming, plugins, and policy. Compare gateway overhead independently from model generation time, then inspect full-task latency.

A fast gateway can still carry expensive context. A compressor can reduce input while adding recovery work. Measure each claim at the layer where it occurs.

## Make migration reversible

Keep the original gateway and configuration available. Add one staging caller, then a bounded production cohort after the required checks pass. Preserve the previous endpoint, alias, credentials, and policy settings.

Use [the gateway migration guide](/guides/gateway-migration) for the test matrix and rollback sequence. Choose the candidate that passes your required contracts and improves the operating result you care about.