---
title: "Add Caveman to LiteLLM: setup, checks and rollback"
description: "Use this guide when LiteLLM is already your gateway and you want to reduce repeated context. Begin with one caller. Your existing provider keys, deployment aliases, and budgets can stay in LiteLLM."
canonical: https://caveman.so/switch/litellm
last-updated: 2026-09-07
---

# Add Caveman to LiteLLM: setup, checks and rollback

Use this guide when LiteLLM is already your gateway and you want to reduce repeated context. Begin with one caller. Your existing provider keys, deployment aliases, and budgets can stay in LiteLLM.

There are two request paths. Choose the one that matches where the agent runs. A proxy on a developer laptop cannot be reached through a remote server's `127.0.0.1`.

## Before changing traffic

Save a copy of the LiteLLM configuration and the caller's current base URL. Record the model alias and authentication method. This integration needs API-key-backed requests; a Claude Pro or Max session cannot authenticate through a LiteLLM virtual key.

Install Caveman on the host that will run its loopback proxy:

```bash
npm install -g @caveman-ai/cli
caveman setup --install
caveman setup
```

For a controlled compression experiment, set the mode explicitly before starting a new listener:

```bash
caveman tools config set think.mode compress
caveman start
```

The listener binds to `127.0.0.1:8787`. If a listener already serves other work, inspect it before changing shared settings. Use a separate test host when you cannot isolate the experiment.

## Path 1: one LiteLLM deployment uses Caveman

This fits an application whose LiteLLM server runs on the same host as Caveman. Add a separate test alias to your existing config; leave the production alias intact.

```yaml
model_list:
  - model_name: caveman-test
    litellm_params:
      model: openai/gpt-5.5
      api_base: http://127.0.0.1:8787/openai/v1
      api_key: os.environ/OPENAI_API_KEY
```

The model above is an example. Use the same supported model as your baseline and an account with access to it. The provider key stays in an environment variable. Point only the test caller at `caveman-test`.

A container has its own loopback namespace. Run both services in the same network namespace or follow the documented same-pod arrangement. Do not solve reachability by exposing Caveman's local proxy publicly.

Recovery on this path is proxy-side. Eligible non-streaming API-key calls can be compressed; streaming sessions pass through when recovery cannot be handled safely. Test the mode your application actually uses before expecting a reduction.

## Path 2: a local agent uses LiteLLM through Caveman

This puts Caveman on the agent's machine. Add a compatibility mount to your Caveman config, merging it with existing entries:

```yaml
# ~/.caveman/caveman.yaml
compat:
  litellm:
    base_url: http://litellm.internal:4000
    api_key_env: LITELLM_VIRTUAL_KEY
```

Supply `LITELLM_VIRTUAL_KEY` through your existing secret mechanism. Replace `litellm.internal` with your gateway host. Private upstreams require an explicit host allowlist.

```bash
caveman tools mcp install claude --server caveman
CAVE_SSRF_ALLOWLIST=litellm.internal caveman start
ANTHROPIC_BASE_URL=http://127.0.0.1:8787/compat/litellm claude
```

Run the listener command in one terminal and Claude Code in another. Configure Claude Code to request an alias that your LiteLLM key permits. Claude appends `/v1/messages`; the compatibility mount forwards the Anthropic-format request. Keep the native recovery tool enabled so a smaller tool result can be expanded when needed.

## Validate before widening the route

Run one task with a known answer against the original configuration, then the same task through the test path. A long log with one specific error code makes a useful first check. Follow it with your normal code-change task and tests.

Confirm that the requested alias reached the intended provider, streaming completed where applicable, and tool calls returned valid results. Force one recovery request and compare the recovered original with the source bytes. If no transform applied, record pass-through; do not count it as a successful compression example.

Inspect local reporting:

```bash
caveman stats
```

Compare those inferred reductions with LiteLLM's provider usage for the entire task. Count recovery turns and failures. The [measurement guide](/guides/measure-agent-cost) explains how to keep cache hits and retries in the comparison.

## Roll back

For path 1, point the caller back to its original model alias. Remove the test alias only after no clients use it. For path 2, close the test agent session and start a fresh session with its saved LiteLLM base URL and authentication settings.

Restore any mode setting you changed on a shared host. Remove the test compatibility entry only after confirming nothing else uses it. Keep logs from both arms long enough to explain the outcome.

## Add Platform separately

Native LiteLLM trace ingestion is a separate private-access Platform integration. It does not require this compression hop. Read [the LiteLLM comparison](/compare/litellm) if the immediate goal is usage analysis or investigations, and agree on content retention before exporting message bodies.

Sources: [Caveman proxy configuration](https://docs.caveman.so/docs/proxy/configure), [proxy security](https://docs.caveman.so/docs/proxy/security), and [LiteLLM documentation](https://docs.litellm.ai/).