---
title: "How to reduce LLM costs in an agent that already works"
description: "Start with the cost of getting useful work finished. A low token price can hide an agent that retries constantly. A long prompt can be cheap when most of it is cached. An inexpensive failed attempt st"
canonical: https://caveman.so/guides/reduce-llm-costs
last-updated: 2026-09-07
---

# How to reduce LLM costs in an agent that already works

Start with the cost of getting useful work finished. A low token price can hide an agent that retries constantly. A long prompt can be cheap when most of it is cached. An inexpensive failed attempt still contributes to the cost of the eventual result.

This guide helps choose the first change to test, then gives you a way to decide whether it belongs in your default setup.

## Find the expensive task family

Group usage by a unit people recognize: a support case resolved, a pull request prepared, an extraction completed, or a report accepted. Include failed attempts and repair runs.

Keep task ownership, environment, model, provider, and date with the record. Unknown prices and missing attribution should remain visible. Do not turn incomplete data into an apparently precise team total.

If you lack task identifiers, begin with a small sample whose boundaries you can reconstruct. A trustworthy sample is more useful than a dashboard that silently mixes unrelated work.

## Choose a change from the evidence

| What you observe | First experiment |
| --- | --- |
| Same large prefix sent repeatedly with poor reuse | Inspect provider cache configuration and prefix stability |
| Long tool results repeated across turns | Test recoverable compression or smaller tool responses |
| Verbose final answers nobody needs | Try shorter response instructions with clarity checks |
| Easy tasks always use the strongest model | Evaluate a smaller model on held-out cases |
| Many tool or schema repair attempts | Fix the tool contract or validation feedback |
| Several agents repeat the same research | Change handoff and ownership before adding more agents |
| Repeated identical requests | Evaluate response caching with correct access and freshness rules |

Do not apply all of these at once. Pick the row supported by the largest reliable cost or failure signal.

## Make tools return what the task needs

A search tool that returns an entire document collection creates work for the model. Prefer useful filters, pagination, projections, and bounded results where the tool's contract allows them.

Keep important exceptions visible. For example, a log tool should not hide the only error line while retaining hundreds of routine health checks. A database tool should distinguish “first page” from “complete result.”

When the next question may need omitted content, preserve a retrieval path. Caveman's [recoverable compression](/guides/prompt-compression) is one option for eligible content. Application-level pagination or a precise follow-up query may be better for structured data you control.

## Preserve good cache reuse

Inspect provider-reported cache reads and writes before shortening a stable prompt. Rewriting earlier turns or tool definitions may reduce reuse.

Run cold and warm trials separately. Include cache creation and storage where applicable, then price the whole task with current rates or charges. A smaller uncached request can cost more than a larger cached request.

The [prompt-caching guide](/guides/prompt-caching) explains the experiment and the different cache layers. Keep gateway response-cache hits separate from provider prefix reuse.

## Test a cheaper model against real failures

Use tasks with known outcomes, including the cases where your current model earns its cost. Keep tool definitions and output requirements fixed. A model that writes valid JSON but chooses the wrong account has failed.

Measure extra turns and fallback calls. If a cheaper first attempt frequently escalates to the expensive model, both calls belong in the total.

Begin with a fixed candidate model before introducing a router. Once you know which tasks each model handles, [model routing](/guides/model-routing) becomes an evidence-based selection problem instead of a guess from a leaderboard.

## Add local Caveman to a checked coding task

For a supported coding agent:

```bash
npm install -g @caveman-ai/cli
caveman setup --install
caveman doctor claude
caveman claude
```

Run the same task through the original launcher in another fresh workspace. Keep the prompt, model, reasoning setting, and relevant tools fixed. Check the patch and run the required tests.

If you only want shorter replies, install and enable the skill through [the coding-agent guide](/guides/coding-agent-setup). Treat that as a separate change from input compression.

Read local observations with:

```bash
caveman stats --json
```

Those reports are inferred. Use provider usage and the complete task record to assess cost; do not convert a local reduction into a claimed subscription saving.

## Fix retries before rewarding them

Inspect repeated tool errors, schema mismatches, and loops that produce no progress. A shorter prompt can make a broken workflow cheaper per turn while leaving the main waste intact.

Give tools clear error contracts and the information needed for one useful correction. Keep side-effecting operations idempotent where your application requires it. Evaluate the failure path with disposable fixtures, then confirm the agent stops or recovers as intended.

When several agents are involved, track the whole task across handoffs. Count failed branches and duplicated research.

## Adopt the change where it passes

Use the same independent acceptance test for baseline and candidate. Report sample count, variation, losing cases, and total cost across all attempts. If no task completes, cost per completion is undefined.

Roll out to the task family that passed. Keep the direct path available and repeat the comparison after meaningful changes to model, agent, tool output, or provider behavior.

[Measure agent cost](/guides/measure-agent-cost) gives the result format. [Agent evaluations](/guides/agent-evaluations) helps build the acceptance suite, and [the comparison index](/compare) maps tools to the layer you need to change.