---
title: "Evaluate Caveman alongside Not Diamond routing"
description: "Keep your Not Diamond route active while preparing a Caveman candidate. Caveman Router requires private access, so confirm the endpoint and project permission before changing an application's default "
canonical: https://caveman.so/switch/not-diamond
last-updated: 2026-09-07
---

# Evaluate Caveman alongside Not Diamond routing

Keep your Not Diamond route active while preparing a Caveman candidate. Caveman Router requires private access, so confirm the endpoint and project permission before changing an application's default path. Without that access, you can still run a local compression trial on the fixed model.

The migration unit is a routing policy for a known workload. It is not a renamed API key.

## Freeze the current Not Diamond policy

Record the model candidates, optimization preferences, custom-router identifier if used, and prompt versions. Save a representative task set that includes the requests your application handles badly as well as the routine ones.

If Not Diamond is also optimizing prompts, save the exact resolved prompt for each test. A new router and a new prompt are two changes. Hold one fixed while evaluating the other.

Keep provider credentials and model access in their existing secret system. A routing service credential does not grant access to every candidate model.

## Set the acceptance criteria

Define a completed task and a maximum tolerated quality regression before looking at results. For a coding agent, include the final patch and tests. For structured extraction, check schema validity and exact fields. Add a latency limit that matches the user-facing workflow.

Use three arms: the current policy, one fixed capable model, and the candidate. The fixed model tells you whether either router is earning its own cost.

Record cache state and run order. A later continuation with a warm prefix is not equivalent to a new conversation. Include any prompt text sent to the router in your data-use review.

## Prepare a LiteLLM decision pilot

Caveman's documented LiteLLM integration asks for a model decision while LiteLLM keeps inference and provider keys. Install the integration from its source package directory in the Python environment that runs your test LiteLLM server:

```bash
python -m pip install .
caveman-router setup --config config.yaml
caveman-router check --config config.caveman.yaml
```

The first command assumes you are already in the `caveman-router` package directory provided for your private-access pilot. Supply the provisioned `CAVE_API_KEY` through your secret environment, with the required `router:write` permission. Do not start this path before access is confirmed.

Review `config.caveman.yaml` against the original. Preserve existing callbacks and require at least two distinct eligible chat models. Run the test proxy with the generated configuration only after the check passes:

```bash
litellm --config config.caveman.yaml
```

This is an alternative policy path. It does not import a Not Diamond custom model or training state.

## Verify abstention and authorization

The Caveman callback supplied for the pilot has eligibility limits. Requests with tools, structured-output controls, reasoning settings, or provider pins can retain their requested model. That is expected behavior, not proof that the router selected a candidate.

Test a timeout and a response naming a model outside the allowed pool in an isolated harness. The request should preserve its permitted baseline route according to the documented callback contract. Keep the virtual key's model restrictions in the experiment.

For a tool-heavy workload that is outside the callback's routing scope, use a fixed route and test local compression instead:

```bash
npm install -g @caveman-ai/cli
caveman setup --install
caveman claude
```

Treat that as a compression experiment, not a replacement routing result.

## Measure full tasks

Count router overhead, provider calls, retries, and failed attempts. Keep both cold and warm sessions. A cheaper first response that causes another debugging cycle belongs in the total cost.

Use [the measurement guide](/guides/measure-agent-cost) and inspect the losing cases. Do not apply a routing threshold trained on one task set as if it were validated on another.

## Roll back

Restore the original LiteLLM configuration or Not Diamond client path for new sessions. Keep in-flight sessions pinned to a known route until they finish. Restore the saved prompt version as well if the experiment changed it.

Sources: [Not Diamond](https://docs.notdiamond.ai/docs/what-is-not-diamond), [Caveman Router](/products/router), and [the comparison](/compare/not-diamond).