---
title: "Model routing for AI agents: quality, cache, and full-task cost"
description: "Model routing chooses which model handles a request or task. A useful router lowers the cost of accepted work while preserving required capabilities, quality, and policy. Choosing the lowest input-tok"
canonical: https://caveman.so/guides/model-routing
last-updated: 2026-09-07
---

# Model routing for AI agents: quality, cache, and full-task cost

Model routing chooses which model handles a request or task. A useful router lowers the cost of accepted work while preserving required capabilities, quality, and policy. Choosing the lowest input-token price is only one small part of that decision.

Before adding a router, prove that at least two models can do useful work in your application. A fixed-model comparison makes later routing results easier to explain.

## Separate eligibility from preference

First decide which models may handle the request. Then choose among them.

| Eligibility check | Why it comes first |
| --- | --- |
| Tool support | The agent must be able to execute its required tool path |
| Structured output | The response must satisfy the application's contract |
| Context and output limits | The request and expected answer must fit |
| Provider and region restrictions | A cheaper route must still obey the caller's policy |
| Credential and model access | A model in a catalog is not necessarily available to this key |
| Reasoning and multimodal needs | Request features must survive the route |
| Session continuity | A model change may affect history, cache, or provider-specific state |

Do not repair an ineligible route by dropping required fields. Keep a known permitted baseline for requests the router cannot safely evaluate.

## Establish fixed-model baselines

Choose representative tasks and run each candidate model with the same tools, starting state, and acceptance tests. Record complete usage, latency, retries, and failures.

Include difficult cases and tasks that require an exact earlier detail. A model may answer a short benchmark correctly while struggling with your tool schemas or long workflow.

Keep the tuning set separate from the final acceptance set. Use [agent evaluations](/guides/agent-evaluations) to define that split and the quality floor.

## Choose a routing unit

Routing each call offers flexibility but can disrupt a multi-turn session. Routing the whole task preserves more continuity but may keep an expensive model on easy intermediate work.

Start at a boundary your application can identify reliably. A task type or explicit capability requirement is easier to audit than an unexplained confidence score. Record why the route was eligible and what happened when it was not.

If the policy uses an embedding model or a separate LLM call, include that work in cost and latency. The router's own computation is part of the system you are evaluating.

## Price fallbacks as additional work

A cheap first attempt followed by an expensive fallback costs more than the fallback alone. Whether that strategy pays depends on how often the first attempt succeeds and how much work it adds when it fails.

```text
attempt cost = routing cost + selected-model calls + fallback calls + repair calls
```

Use accepted completions as the outcome. A route that refuses all difficult work can look cheap while failing the product's purpose.

Decide which component owns retries and fallbacks. A client, gateway, and router each retrying independently can create more provider calls than the application expects.

## Keep cache effects inside the comparison

Changing model or provider can lose a warm prefix. A lower list price may not compensate for that loss during a long agent session.

Compare first-use and repeated-context tasks. Preserve provider cache-read and cache-write evidence, and record whether the route moved mid-session. [The caching guide](/guides/prompt-caching) explains why token reduction and price reduction can diverge.

Compression should be a separate experiment at first. Hold the route fixed, change eligible context, and check the complete result. Then test the combined policy if both changes have evidence behind them.

## Evaluate the available approaches

[RouteLLM](/compare/routellm) provides an open-source routing experiment with explicit methods and thresholds. [Not Diamond](/compare/not-diamond) offers routing and optimization products. [Martian](/compare/martian) has its own routing approach and integration paths. Gateways such as [LiteLLM](/compare/litellm) and [OpenRouter](/compare/openrouter) also own provider and fallback behavior you must account for.

Caveman Router is available through private access. Its LiteLLM integration keeps inference and provider keys in LiteLLM while adding an eligible model-selection path. The current callback can retain the original route for request shapes such as tools, structured output, reasoning, or provider pins. Do not assume a routing pilot covers every agent request.

Review [Caveman for LiteLLM](/switch/litellm) and use the configuration supplied for your pilot. There is no public anonymous Router endpoint to paste into any SDK.

## Start with observation and a bounded candidate

Keep the original route callable. Evaluate the candidate on held-out tasks before changing production defaults. Record the eligible pool, selected model, baseline, decision latency, failure behavior, and final result.

For a live pilot, choose a limited task family and a clear rollback trigger. Test unavailable models, decision timeouts, missing usage, and a caller that lacks permission for one candidate. The result should preserve the caller's contract.

## Recalibrate when the system changes

A threshold belongs to a model pair, dataset, and deployment context. New models, changed prompts, or different tool output can invalidate earlier results.

Repeat the fixed baselines and acceptance suite when those conditions change. Use [cost per completed task](/guides/measure-agent-cost) as the economic result, and keep quality, latency, and policy failures visible beside it.