---
title: "Caveman vs LiteLLM: keep your gateway, cut agent waste"
description: "Keep LiteLLM if it already handles your provider keys and team budgets. Add Caveman when your coding agents repeatedly send large tool results back to a model, or when you want to investigate the work"
canonical: https://caveman.so/compare/litellm
last-updated: 2026-09-07
---

# Caveman vs LiteLLM: keep your gateway, cut agent waste

Keep LiteLLM if it already handles your provider keys and team budgets. Add Caveman when your coding agents repeatedly send large tool results back to a model, or when you want to investigate the work behind the spend. You can try local compression without migrating your gateway.

[LiteLLM](https://docs.litellm.ai/) provides a shared provider API through a Python library or proxy server. Caveman's local proxy reduces eligible context before inference and keeps the original bytes available for recovery. Caveman Platform adds a separate path for trace analysis; it currently requires private access.

## What each tool owns

| Decision | LiteLLM | Caveman |
| --- | --- | --- |
| Provider access | Provider translation, virtual keys, retries and budgets | Local proxy can forward to LiteLLM; keys and inference can stay there |
| Repeated tool output | Compression can be added through integrations such as Headroom | Built-in content-specific compressors with local recovery handles |
| Usage analysis | Spend tracking at the gateway | Local token reports, plus native LiteLLM trace ingestion in Platform |
| Model choice | Deployment selection and routing controls | Optional routing integration, evaluated separately from compression |
| First experiment | An existing deployment and virtual key | One wrapped agent or one local deployment path |

LiteLLM also supports caching and guardrails. Those are real cost and reliability controls. A comparison that gives Caveman every checkmark and LiteLLM only a provider list would miss why teams adopted it.

## Where Caveman earns its place

A gateway can report that a request used many input tokens. It does not follow that changing gateways will remove those tokens. If your agent has read a long log or returned a large JSON object, the content itself needs attention.

Caveman stores the original locally before returning a smaller view with a recovery handle. A failed parse, failed store write, or candidate that is not smaller keeps the original. This is useful for tool-heavy tasks where the model needs the ability to look back at detail.

Start with that change alone. Keep the requested model and LiteLLM policy fixed so you can see whether compression helped. Changing the compressor, model, and gateway together leaves you guessing about the cause.

## Two ways to connect them

For a local coding agent, the request path can be agent → Caveman → LiteLLM → provider. The agent needs a working Caveman recovery tool. The compatibility mount uses a LiteLLM virtual key and an explicitly allowed upstream host.

For an application already calling LiteLLM, a deployment can instead point at Caveman on the same host. That path has a narrower recovery contract: eligible non-streaming API-key calls can use proxy-side recovery; streaming passes through when safe recovery is unavailable. Do not assume that adding the hop compresses every request.

The [switching guide](/switch/litellm) covers both paths, the loopback constraint, and how to restore the original route.

## Traces without moving inference

Caveman Platform can receive native LiteLLM traces while provider traffic remains in LiteLLM. That gives a team a route into usage analysis and investigations without replacing the gateway. Retained message content is a separate choice from metadata ingestion.

This is a private-access integration. Use the [LiteLLM switching guide](/switch/litellm) to plan the integration and [contact Caveman](/contact) for a pilot. Local compression does not require a Platform account.

## Compare the result that matters

Keep one task set and one correctness check. Include failed attempts and recovery calls. Record fresh input, cache reads and writes, output, elapsed time, and the final answer or patch result. A smaller request is useful only if the whole task improves.

Local Caveman reports are labelled `inferred`. LiteLLM's provider usage records and your provider invoice answer different questions from the local reduction estimate. Do not add the two together as separate savings.

If your current problem is key distribution, provider coverage, or rate-limit handling, work on the LiteLLM configuration first. If repeated agent context is the problem, Caveman gives you a specific change to test without taking that setup apart.

## Sources and next steps

- [LiteLLM overview](https://docs.litellm.ai/): proxy, SDK, provider translation and gateway controls.
- [LiteLLM cost guide](https://docs.litellm.ai/blog/save-claude-code-costs-with-litellm): compression alongside other cost controls.
- [Caveman with LiteLLM](/switch/litellm): configuration and recovery limits.
- [Measure an agent optimization](/guides/measure-agent-cost): a repeatable comparison method.