---
title: "Caveman vs RouteLLM for cost-aware model routing"
description: "RouteLLM is useful when you want to serve and evaluate routing methods yourself. Caveman is useful when the question is how an optimization behaves inside an agent workflow, with local compression ava"
canonical: https://caveman.so/compare/routellm
last-updated: 2026-09-07
---

# Caveman vs RouteLLM for cost-aware model routing

RouteLLM is useful when you want to serve and evaluate routing methods yourself. Caveman is useful when the question is how an optimization behaves inside an agent workflow, with local compression available now and routing through private-access pilots.

The distinction is operational. A good router benchmark helps you choose a policy. A production agent also needs to preserve tool behavior, permissions, and the cost of continuing the task after that choice.

## Research framework and application policy

[RouteLLM](https://github.com/lm-sys/RouteLLM) provides trained routers, an OpenAI-compatible serving path, evaluation tooling, and threshold calibration. Its documented setup commonly routes between a stronger and weaker model. Those are useful baselines for a cost-quality experiment.

| Decision | RouteLLM | Caveman |
| --- | --- | --- |
| Router methods | Framework for serving and evaluating different routers | Product routing evaluated against a workload contract |
| Cost-quality control | Calibrated threshold and model pair | Eligible candidates, required quality, and session economics |
| Existing gateway | Integrate the serving or controller path | LiteLLM callback can request a decision without owning inference |
| Non-routing optimization | Build separately around the application | Local context compression and recovery |
| Rollout availability | Public framework | Router in private development |

## Where a benchmark stops

A question-answer benchmark can establish useful model preferences. It may not cover an agent that writes a patch, runs tests, sees a failure, and resumes. That trajectory contains paid work after the first answer.

Switching models midway can change how the agent uses tools or interprets earlier context. A new route can also lose a useful cache prefix. Include these effects when deciding whether the lower token price helped.

Caveman's routing evaluation is aimed at that total-task question. This page does not turn that aim into a measured superiority claim. Compare actual candidates against your fixed-model and RouteLLM baselines.

## Keep calibration honest

A threshold only means something with its router, model pair, and calibration data. Reusing a published threshold with different models is an experiment, not a preserved quality guarantee.

Save the calibration set and evaluate on different tasks. Include hard cases where the smaller model is likely to fail and easy cases where the strong model wastes money. The candidate pool must satisfy capability and access requirements before a router ranks it.

Tools and structured outputs need explicit tests. A request can be valid for a chat model and incompatible with the tool protocol your agent requires.

## Caveman without replacing RouteLLM

If you already have a routing policy that works, test whether reducing repeated context helps on the selected model. Caveman's local wrapper or engine gives you that additional optimization without retraining the router.

Keep the two experiments separate initially. A smaller prompt can change a router's decision, and then your comparison no longer isolates compression. Freeze the route for the first trial and inspect the result using [cost per completed task](/guides/measure-agent-cost).

## When to evaluate a switch

Consider a Caveman pilot when keeping inference in an existing gateway or measuring long-session economics is more useful than operating a research router directly. Keep RouteLLM when control over the routing method and calibration pipeline is central to your work.

The [switching guide](/switch/routellm) preserves the current controller or server setup, builds a comparable model pool, and defines rollout and rollback checks. The [routing guide](/guides/model-routing) explains the common evaluation method.

Sources: [RouteLLM repository](https://github.com/lm-sys/RouteLLM), [Caveman Router](/products/router), and [Caveman LiteLLM integration](/switch/litellm).