---
title: "How Caveman works with LiteLLM"
description: "Running LiteLLM for your team? [Connect it to Caveman Platform](/solutions/litellm)"
canonical: https://caveman.so/news/caveman-and-litellm
last-updated: 2026-09-15
---

# How Caveman works with LiteLLM

**Running LiteLLM for your team? [Connect it to Caveman Platform](/solutions/litellm)
through OpenTelemetry.** Keep your existing gateway and provider keys. Bring
traces, reported usage, and optional retained messages into Caveman for debugging,
evaluations, and investigations. The local proxy options below are separate ways
to add compression; they are not required for the platform connection.

LiteLLM is a gateway. It holds provider keys, maps a hundred provider APIs
onto one shape, enforces budgets per virtual key, and retries. Caveman is an
efficiency layer. It shortens what an agent writes, shrinks what it rereads,
and measures the difference. The two do not compete for the same job, and
the seams between them are small.

Start with the platform connection if you run LiteLLM for a team. The local
skill and compression options below work independently. Platform traces and
model routing use a Caveman project key.

## Connect Caveman Platform through native traces

Your inference path stays `app → LiteLLM → provider`. Enable LiteLLM's native
OpenTelemetry exporter, or add Caveman to your existing collector's export
pipeline. Caveman receives the trace tree, models, token counts, timing,
errors, and reported cost without becoming another inference hop.

Opt into message and tool-content capture when you need retained evidence for
evaluation datasets, scenarios, and investigations. Capture requires storage
and purpose consent; redaction, encryption, access controls, and retention
apply. Metadata export works without content capture.

[Connect LiteLLM to Caveman Platform](https://app.caveman.so/integrations/litellm)
for copyable settings and a live connection check. The setup is tested with
LiteLLM 1.96.0, including streamed requests and provider errors. Trace export
alone does not apply compression or prove savings.

## What LiteLLM and the local Caveman tools own

| Concern | LiteLLM | Caveman |
| --- | --- | --- |
| Provider credentials | Holds them | Never sees them in local mode |
| Provider API translation | Yes, 100+ providers | Native shapes only: Anthropic, OpenAI, Gemini, Bedrock, Azure, Vertex |
| Budgets, virtual keys, teams | Yes | No |
| Retries, fallbacks, load balancing | Yes | No |
| Output length | No | The skill, in the agent |
| Tool-output compression with recovery | No | The local proxy |
| Model decision per request | Rule and embedding routers | The `caveman-router` callback |
| Measurement basis | Provider usage per key | Provider usage plus local inferred estimates, kept separate |

For local compression, the optional Caveman proxy sits between the agent
and LiteLLM. The platform connection above sends telemetry separately.

## 1. The skill needs nothing from the gateway

The [Caveman skill](https://docs.caveman.so/docs/skill) is a rule file the
agent loads. It changes how Claude, Codex or Gemini writes, so it is upstream
of any gateway. Install it in the agent and LiteLLM never knows.

```bash
npx skills add JuliusBrussee/caveman
```

On ten measured coding prompts it cut output tokens by 65 percent. It adds
roughly 1,000 to 1,500 input tokens per turn for its own rules, so on already
terse sessions it can cost more than it saves. That trade does not change
when a gateway is in the path.

## 2. Per call: point `api_base` at the local proxy

If your Python code calls `litellm.completion` directly, the smallest change
is one argument. Start the local proxy, then aim the call at it.

```bash
npm install -g @caveman-ai/cli && caveman setup --install
caveman tools config set think.mode compress   # bare start defaults to record
caveman start                                  # listens on 127.0.0.1:8787
```

```python
import os
import litellm

res = litellm.completion(
    model="openai/gpt-5.5",
    api_base="http://127.0.0.1:8787/openai/v1",
    api_key=os.environ["OPENAI_API_KEY"],
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
```

For Anthropic models the mount is `/anthropic/v1`, for Gemini `/gemini/v1beta`.
The [providers page](https://docs.caveman.so/docs/proxy/providers) lists every
route. The key travels on the request and the proxy forwards it. Nothing is
stored except the originals of compressed payloads, in a local SQLite file, so
the agent can recover exact bytes later.

The CLI prints this snippet for you, with the right mount for your provider:

```bash
caveman snippets litellm
```

## 3. Fleet-wide: put the proxy in `config.yaml`

If you run the LiteLLM proxy server, every client behind it inherits whatever
you set on a deployment. Point the deployment at Caveman instead of the
provider and compression applies to all of them.

```yaml
model_list:
  - model_name: gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_base: http://127.0.0.1:8787/openai/v1
      api_key: os.environ/OPENAI_API_KEY
```

Two things to know before you do this on a shared host.

The Caveman proxy listens on loopback only. Run it on the same machine as the
LiteLLM proxy, or in the same pod. It refuses non-local upstreams unless you
add an exact host to its allowlist, which is a security property, not a
limitation to work around.

Recovery has to live somewhere. When the agent runs on the same machine, it
recovers originals through the Caveman MCP tool. A client on another machine
behind LiteLLM has no such tool, so the proxy compresses only where it can
handle recovery itself, which today means non-streaming API-key requests on
supported adapters. Streaming sessions on this path go through unchanged.
The [wrap page](https://docs.caveman.so/docs/proxy/wrap) has the exact rule.

Compression runs before LiteLLM's own budget accounting sees the response.
LiteLLM will count what the provider reported, which is the compressed
request. That is correct behaviour. It means your LiteLLM spend dashboard
shows the post-compression bill, and the local `caveman stats` shows what
was removed, labelled as an inferred estimate.

## 4. Claude Code through LiteLLM through Caveman

Teams often run Claude Code through LiteLLM already, for budgets and one set
of keys. Adding Caveman keeps that arrangement and puts compression at the
front.

```text
Claude Code  →  Caveman proxy (loopback)  →  LiteLLM proxy  →  Anthropic
```

Declare your LiteLLM host as a compatibility mount, allow it as an upstream,
give Claude Code the recovery tool, and point Claude Code at the mount:

```yaml
# ~/.caveman/caveman.yaml
compat:
  litellm:
    base_url: http://litellm.internal:4000
    api_key_env: LITELLM_VIRTUAL_KEY
```

```bash
caveman tools config set think.mode compress
caveman tools mcp install claude --server caveman
CAVE_SSRF_ALLOWLIST=litellm.internal caveman start
ANTHROPIC_BASE_URL=http://127.0.0.1:8787/compat/litellm claude
```

Claude Code appends `/v1/messages`, and a compat mount carries Anthropic
protocol on that path with the key in `x-api-key`. Caveman compresses tool
results in Messages shape, then forwards to LiteLLM, which serves its
Anthropic-format endpoint as it normally would. Your virtual key still
enforces its budget window. Your cache breakpoints pass through untouched,
because Caveman never rewrites the `cache_control` markers a client sent.

One limit: a Claude Pro or Max login cannot cross LiteLLM. This path needs an
API-key backed virtual key. If you want subscription auth, drop LiteLLM from
the chain and run `caveman claude`, which passes the OAuth token straight to
Anthropic.

Measured on the pinned benchmark, direct against wrapped, Claude Code used
33.2 percent fewer provider-reported input tokens across 18 paired runs with
all 18 exact-answer checks passing. One case, an HTML dashboard, got 9.9
percent more expensive because no compressor applied. The gateway hop does
not change either number.

## 5. Model decisions inside LiteLLM: `caveman-router`

Everything above moves bytes. This section moves a decision.

`caveman-router` is a small MIT Python package that registers as a LiteLLM
callback. On each chat request it asks Caveman which of your configured
deployment aliases to use, then changes only the alias. LiteLLM keeps the
provider keys, does the inference, streams the response, and handles retries
exactly as before. Inference never passes through Caveman.

```bash
# in the Python environment that runs your LiteLLM proxy
python -m pip install .           # from the caveman-router package directory
export CAVE_API_KEY='project key with router:write'
caveman-router setup --config config.yaml
caveman-router check --config config.caveman.yaml
litellm --config config.caveman.yaml
```

`setup` writes a second file beside your existing one. It adds the callback
first in `litellm_settings.callbacks`, so later callbacks see the chosen
alias, and it leaves every provider setting alone. It discovers eligible chat
deployments from `model_list` and refuses to proceed with fewer than two
distinct models, because a router with one option is not a router.

What the callback will not touch:

- Requests with `tools`, `response_format`, `reasoning_effort`, `thinking`,
  audio, `logit_bias`, or a `previous_response_id`. Those depend on model
  capabilities the decision does not carry, so the request is left as sent.
- Requests that pin a provider: `api_base`, `api_key`, `custom_llm_provider`,
  Azure, Bedrock or Vertex fields, or an explicit deployment.
- Anything outside the models the virtual key or team is allowed to call.
  The callback intersects with LiteLLM's own scope before asking, then
  re-runs LiteLLM's authorization check on the answer.
- Any model whose context window or supported parameters, read from LiteLLM's
  local catalog, cannot hold the request.

The decision has a wall-clock deadline, 150 milliseconds by default. On a
timeout, an HTTP error, or an unparseable answer, the request proceeds on the
model the client asked for and the callback logs one warning. Prompts are
truncated to 8 KB before they leave, tool results are omitted, and no headers
or tenant identifiers are sent.

The decision service behind `/v1/route` is part of Caveman Platform. The
package, its setup tool, and its conformance tests against a real LiteLLM
server are public. If you have a Platform project key, it works as described;
if you do not, the first four sections stand on their own.

## Headroom inside LiteLLM

LiteLLM's guide recommends Headroom as its compression sidecar. We benchmarked
both against direct Claude Code on the same six tool-heavy tasks, three runs
each, with an exact-answer check on every run.

| Arm | Exact answers | Input tokens | Reduction |
| --- | ---: | ---: | ---: |
| Direct Claude Code | 18 of 18 | 885,793 | baseline |
| Caveman wrap plus skill | 18 of 18 | 591,673 | 33.2 percent |
| Headroom wrap | 15 of 18 | 703,202 | 6.7 percent |

Headroom's three YAML runs failed the exact-answer gate and stay visible in
its row rather than counting toward savings. Caveman's HTML case lost 9.9
percent and stays visible in ours. Method, intervals, and fixtures are in the
[benchmark write-up](https://github.com/JuliusBrussee/caveman/blob/main/docs/WRAP-BENCHMARK.md).
Both tools slot into the same place in a LiteLLM deployment, so switching is
a config change.

## Questions people ask

### Does Caveman replace LiteLLM?

No. LiteLLM holds provider keys, translates provider APIs, and enforces budgets
per virtual key. Caveman shortens output, compresses tool results with
recovery, and can supply a model decision through a callback. Run both. Put
Caveman between the agent and LiteLLM.

### Does compression break LiteLLM's spend tracking?

No. LiteLLM records what the provider reported, which is the compressed
request. Its dashboard shows the bill you actually paid. The amount removed is
reported separately by `caveman stats` as an inferred local estimate, and the
two are never added together.

### Can the router callback see my provider keys?

No. The callback changes the deployment alias on the request and nothing
else. Keys stay in LiteLLM's config. The callback sends at most 8 KB of
prompt text and the list of allowed aliases to the decision endpoint, with
no headers, tool results, or tenant identifiers.

### What happens if the Caveman decision service is down?

The request goes to the model the client asked for. The callback has a 150
millisecond deadline by default and treats a timeout, an HTTP error, or an
invalid response the same way: keep the requested model, log one warning,
count the reason.

### Does the local proxy work with LiteLLM's prompt caching injection?

Yes. Caveman forwards `cache_control` markers unchanged. If LiteLLM injects
them, they arrive at the provider as injected. If the client sends them, they
arrive as sent. Compression changes tool-result bodies, which sit after the
cached prefix in a normal agent request, so a stable prefix stays stable.

## Sources

- [LiteLLM integration recipe](https://github.com/JuliusBrussee/caveman/tree/main/integrations/recipes) and `caveman snippets litellm`.
- [Proxy providers and mounts](https://docs.caveman.so/docs/proxy/providers), [security](https://docs.caveman.so/docs/proxy/security).
- [CaveBench Wrap benchmark](https://github.com/JuliusBrussee/caveman/blob/main/docs/WRAP-BENCHMARK.md), 54 runs.
- [LiteLLM, 5 ways to cut Claude Code costs](https://docs.litellm.ai/blog/save-claude-code-costs-with-litellm).