---
title: "Move a RouteLLM experiment into a Caveman pilot"
description: "Keep the RouteLLM controller or server as a callable baseline. A Caveman pilot needs its own access and configuration, and it does not import RouteLLM model weights or turn an existing threshold into "
canonical: https://caveman.so/switch/routellm
last-updated: 2026-09-07
---

# Move a RouteLLM experiment into a Caveman pilot

Keep the RouteLLM controller or server as a callable baseline. A Caveman pilot needs its own access and configuration, and it does not import RouteLLM model weights or turn an existing threshold into a Caveman policy.

Before moving traffic, decide whether the goal is replacing model selection or adding context compression to a route that already works. The second task can start locally without Router access.

## Save the RouteLLM experiment

Record the router method, threshold, strong and weak model IDs, configuration file, and package revision. Preserve the calibration data and a separate held-out task set. If you serve RouteLLM through its OpenAI-compatible endpoint, save the caller's base URL and routed model string too.

A model string such as `router-mf-0.5` encodes a RouteLLM policy. It is not a destination-provider model ID and should not be sent unchanged to another router or provider.

Include the cost of router dependencies in the baseline. Some methods require embeddings or other computation in addition to the eventual model call. Check the [current RouteLLM instructions](https://github.com/lm-sys/RouteLLM) for the method you use.

## Build a comparable candidate pool

Map each baseline model to an actual deployment alias in your test gateway. Confirm access, protocol support, and the capabilities needed by the task. Keep region or provider restrictions explicit.

Use the same models for the first policy comparison. If the candidate adds newer models, report that as another experiment. Otherwise an apparent routing improvement may be a model upgrade.

Define the fallback before testing. A decision timeout, an unknown alias, or an ineligible request should have a known result that preserves the caller's permissions.

## Configure the Caveman pilot

For a provisioned Caveman project, use the source integration in the Python environment running your test LiteLLM proxy:

```bash
# From the caveman-router source package directory
python -m pip install .
caveman-router setup --config config.yaml
caveman-router check --config config.caveman.yaml
```

Supply the project credential with `router:write` through the environment. Obtain the integration source and installation instructions supplied for your pilot. This setup requires private Router access; it is not a public anonymous endpoint.

Review the generated configuration before starting the candidate:

```bash
litellm --config config.caveman.yaml
```

Provider keys and inference stay in LiteLLM. The callback's eligibility rules are narrower than every possible OpenAI-shaped request. In particular, tool, structured-output, reasoning, or provider-pinned requests can retain their original model. Test the real request type you intend to route.

## Compare policies on unseen tasks

Run the current RouteLLM policy, the Caveman candidate, and a fixed strong-model baseline against the same held-out tasks. Alternate order where practical. Measure the final accepted result rather than only the router's preferred model.

For multi-turn agents, preserve the same starting conversation and workspace. Include cache changes, recovery or repair turns, and total elapsed time. Grade the resulting patch or answer independently of the model's success claim.

Use [the cost measurement guide](/guides/measure-agent-cost) for a result format. If no candidate task passes, the result is a failed trial, not a zero-cost completion.

## Add compression as a separate arm

If repeated context is expensive, hold the model route fixed and test Caveman locally:

```bash
npm install -g @caveman-ai/cli
caveman setup --install
caveman codex
```

Use the appropriate supported launcher for your agent. Verify recovery on a tool-heavy task. This can establish value from Caveman even when the routing policy stays in RouteLLM.

## Roll back

Point new sessions back to the saved RouteLLM controller or server and its original threshold. Keep the baseline environment available throughout the pilot. Remove only the candidate callback and its configuration when no caller uses it.

Retain both task sets and result records. A later model or router update should be compared against the same baseline instead of inheriting an earlier success label.

Read [the comparison](/compare/routellm) and [model-routing guide](/guides/model-routing) for the decision criteria.