---
title: "How to measure whether a token optimization saved anything"
description: "Every tool in this space, ours included, can show you a number that says it"
canonical: https://caveman.so/news/how-to-measure-agent-token-savings
last-updated: 2026-09-15
---

# How to measure whether a token optimization saved anything

Every tool in this space, ours included, can show you a number that says it
saved tokens. Most of those numbers are produced by the tool being measured.
A method that cannot be wrong is not a measurement.

This is the procedure we use for our own benchmarks and the one we would ask
you to use on us. It takes an afternoon for one workload and it settles
arguments that dashboards cannot.

## Why tool counters mislead

Three things we have published, all from our own runs:

- A local token counter reported 96.2 million tokens saved. Paired trials on
  the same work found a higher provider bill. The counter measured bytes
  removed from payloads; the bill measured what the provider charged after
  retries, cache misses and re-injected rules. Same session, opposite sign.
- A benchmark printed a 26.5 percent reduction. Only nine task pairs held
  quality, the bootstrap interval crossed zero, and output tokens had risen
  37 percent. The report marked itself unpublishable and we kept it out of
  every claim.
- A 998-task replay gained seven net passes while 125 individual outcomes
  flipped. The aggregate was flat. The distribution was not.

Each of these looked like an answer for a day. The method below is what
caught them.

## The method

### 1. Pair the tasks

Agent workloads vary enormously. Comparing a hard investigation on one arm
with an easy lookup on the other mostly measures task selection. Run the same
task, from the same starting state, on both arms. Three repetitions per arm
per task is the minimum that gives you a spread.

Rotate the order. Cache state and provider load drift over an afternoon, and
if arm A always goes first it always pays the cold start.

### 2. Use the provider's counters

Take the numbers from the response, or from the agent's own usage log if it
records provider counts. Do not use a local tokenizer. Sum the three input-side
counters:

```text
input_tokens + cache_read_input_tokens + cache_creation_input_tokens
```

Record output tokens separately. Record the cache buckets separately too,
because a change that shifts tokens from fresh input to cache reads is a real
saving in dollars and a zero in raw token count. Weight by price when you have
the rates.

### 3. Put a correctness gate on every run

A cheaper wrong answer is not a saving. Every run needs a check the task can
fail: an exact expected value, a test suite, a structured answer compared to a
key. Runs that fail the gate count as zero savings at held quality. They are
not excluded, because excluding them is how a small result becomes a large
claim.

### 4. Count the overhead

Whatever the optimization adds, count it from the first request. Rule files
injected into the prompt. Recovery calls and the input they trigger. Tool
definitions the optimization installs. If a tool saves tokens on a large
payload and costs tokens on every small one, the session number is the only
honest one.

### 5. Report the spread as well as the mean

Cluster by task and bootstrap the aggregate. If the interval crosses zero,
say so. If one task went negative, show the row. Our pinned Claude Code
benchmark reports 33.2 percent with an interval of 14.6 to 48.5 percent, and
an HTML case at minus 9.9 percent. The spread is the finding.

## Label what you measured

A number without its basis gets promoted the moment someone pastes it into a
slide. We use four labels and do not let them convert into each other by
wording.

| Label | What produced it | What it can claim |
| --- | --- | --- |
| inferred | A local estimate, usually an offline tokenizer against catalog list prices | Direction and rough size. Never a bill. |
| provider-reported | Usage counters returned by the model provider | What was charged, per request. |
| benchmark_counterfactual | Paired runs under a pinned, published method | What the change did on those tasks. Not production. |
| verified | Provider-causal evidence on live traffic, both arms, signed | The only label that may say saved dollars. |

A local estimate stays inferred no matter how plausible it looks. A paired
benchmark stays counterfactual no matter how large. Nothing on your laptop
mints verified.

## Run it with `caveman trial`

The CLI wraps the procedure for a real coding session. It runs the task with
and without the local proxy, keeps provider counts, and prints the paired
comparison with the labels attached.

```bash
caveman trial -- claude
caveman trial report
```

It measures Caveman. The same procedure, by hand, measures anything else in
the path, including a gateway's caching injection or a router's model
choices. If a vendor's number cannot survive it, that is worth knowing before
the invoice does.

## Questions people ask

### Why not trust the token counter in my agent?

Because it usually counts something other than what you pay for. Local
counters estimate with an offline tokenizer, ignore cache pricing, and do not
see retries. We published a case where a counter reported 96.2 million tokens
saved while paired trials showed a higher provider bill.

### How many runs do I need?

Three per arm per task is the floor. It gives you a spread and lets a single
slow provider response show up as noise rather than a result. More is better
on tasks with high variance.

### Should I exclude runs that failed?

No. Count them as zero savings at held quality and leave them in the table.
Excluding failures is the most common way a modest result becomes a large
claim.

### What does verified mean in Caveman?

Provider-causal evidence on live traffic, with both arms observed, signed into
a ledger. Local tools never produce it. Benchmarks never produce it. It starts
at zero dollars and stays there until a change is active on real requests.

### Can I use this method on LiteLLM or another gateway?

Yes. Nothing in it is specific to Caveman. Pair the tasks, take provider
counters, gate on correctness, count overhead, report the spread. Then label
the result by how it was produced.

## Sources

- [How removing tokens raised the bill](/news/how-removing-tokens-raised-the-bill).
- [A benchmark printed 26.5 percent. We refused the result](/news/we-cut-26-percent-and-refused-the-claim).
- [The average stayed flat. 125 tasks changed outcome](/news/the-average-stayed-flat).
- [CaveBench Wrap benchmark](https://github.com/JuliusBrussee/caveman/blob/main/docs/WRAP-BENCHMARK.md), the method in full.
- [How numbers are counted](https://docs.caveman.so/docs/counting), the four labels.