---
title: "How removing tokens raised the bill"
description: "In July, a tool designed to shrink coding-agent output reported that it had"
canonical: https://caveman.so/news/how-removing-tokens-raised-the-bill
last-updated: 2026-08-13
---

# How removing tokens raised the bill

In July, a tool designed to shrink coding-agent output reported that it had
saved 96.2 million tokens. The paired provider bill moved in the other
direction. Across 80 clean task pairs at low reasoning effort, the treatment
cost 7.6 percent more. It also produced 13.8 percent more turns and 14.3 percent
more cache reads.

Both measurements were correct within their own frames: one counted characters
removed when a shell command returned, while the provider counted what it
charged to complete the task. Between those counters sits the agent loop, where
a small edit to one observation can change future actions, invalidate a prompt
cache, or provoke another tool call.

That is the trap in agent-efficiency work. Local tokens are easy to count; cost
belongs to the entire trajectory.

## A block is charged for its future

A chat request has one input and one output. A coding agent has a growing
prefix. Each turn sends the system instructions, tool definitions, conversation
and earlier tool results back to the model. Prompt caching may discount the
stable part, though those tokens still occupy context and still affect what the
model sees.

For a session with `T` model calls, a useful first approximation is:

```text
input work = sum(prefix at turn t), for t = 1..T
```

A 4,000-token test log inserted after turn three of a 30-turn session can be
carried through roughly 27 later calls. Counted once, it is 4,000 tokens.
Counted as context processed over the remaining trajectory, it is about 108,000
token-occurrences before cache pricing.

The same multiplication applies to fixed instructions and tool schemas, making
turn count unusually dangerous. Each extra turn pays for the accumulated prefix
alongside its new response.

That gives any token-reduction claim three separate denominators:

| Counter | What it answers | What it misses |
| --- | --- | --- |
| Local reduction | How much shorter was this block? | Later turns, cache effects, changed behavior |
| Provider usage | What token classes did the API report? | Task value and counterfactual outcome |
| Paired task cost | What did the same task cost under both conditions? | Generalization beyond the tested workload |

The first counter is useful engineering telemetry. It is a weak savings claim.

A compressor grades its output. A paired task grades the system.

## Prompt caching changes the sign

Current Claude pricing makes the cache arithmetic easy to see: a five-minute
cache write costs 1.25 times the base input rate, versus 0.1 times for a cache
hit. One read repays the write premium; later reads make a stable prefix cheaper
still.

Now suppose an optimizer revises an old 10,000-token block after it has entered
that stable prefix. The revised block may be shorter, but the edit breaks exact
prefix identity at that point. The next request must ingest the changed suffix
again. A token removed from the old block saves a future read at 0.1 times the
base rate; the rewrite can trigger fresh ingestion at 1 or 1.25 times base.

The relevant comparison is therefore weighted:

```text
net effect
  = discounted future reads avoided
  - new cache write or cold-input work
  - optimizer call cost
  - cost of any additional agent turns
```

Raw token count is only one term. Expected remaining calls determine whether
timing and prefix location turn a rewrite into a win.

We saw the sign flip in a fixed-trajectory replay over 24 recorded coding
sessions. In one 12-session arm, an aged-context rewrite reduced modeled raw
tokens by 0.75 percent while raising cache-weighted cost by 0.83 percent. In the
other arm, raw tokens fell 0.44 percent and weighted cost rose 2.08 percent.
The culprit was re-ingestion: the rewrite saved bytes and rebuilt cached
suffixes.

Those figures are an inferred mechanism study, not a measured savings result.
The replay held agent behavior fixed, reconstructed 80.2 percent of
provider-reported tokens in one arm and 35.1 percent in the other, and could not
observe the extra turns a lossy edit might cause. Its value is narrower. It
proves that cache arithmetic alone can reverse a token win before behavior even
enters the calculation.

## The model can spend the bytes again

Behavior creates the larger failure mode. A shortened terminal result may omit
the exact test name, path, version or error line that determines the next move.
The agent rarely announces the loss. It searches again, reruns the test, reads
the file, or takes another plausible route. Every recovery action adds a turn
and carries the prefix once more.

JetBrains' paired evaluation of `rtk`, the tool behind the 96.2-million counter,
found a related system-level penalty without isolating this recovery mechanism.
Its transcript review found only a few compression-induced rereads and no single
culprit; ordinary trajectory variance explained much of the movement. The hook
also reached only a subset of shell results. Built-in file reads bypassed it,
many commands sat outside its rules, and Claude Code already truncated extreme
outputs. A free transcript replay put the likely input-token ceiling near 3
percent. Paid trials then found a higher bill, with greater rewrite exposure
correlated with a larger penalty at low effort. Correlation did not identify the
cause.

The tool's local analytics still reported a 99.8 percent reduction across the
content it touched. One `cat` of a 1.2 MB CSV received credit for roughly
320,000 saved tokens even though the harness would have truncated the raw
result before sending it to the model. The counterfactual existed only inside
the tool's accounting.

We learned a related lesson from an independent evaluation of our own Caveman
writing skill. Short chat-style answers had suggested a 65 percent output-token
reduction. JetBrains forced the skill on across 82 clean paired coding tasks and
measured 8.5 percent. Code, diffs, tool calls and error strings dominated the
agent's output, and the skill correctly left them alone. Quality was
statistically indistinguishable. The original percentage described the slice
we had compressed, then escaped its denominator and became a claim about the
whole workload.

[Read how InfoWorld, TechTimes, and ITmedia covered that evaluation.](/press#benchmarks)

That is how honest component measurements turn into bad system claims. Nothing
needs to be fabricated. A denominator merely goes missing.

## Run the cheap experiment first

Before paying for a large paired benchmark, replay existing transcripts and
bound the surface an optimizer can reach. This does not establish savings. It
can kill impossible claims at almost no cost.

Start with five quantities:

1. Total provider-reported input, split into uncached input, cache creation and
   cache reads.
2. Output and reasoning usage.
3. Share of the trajectory the optimizer can touch after harness truncation and
   tool exclusions.
4. Carry depth: how many later model calls receive each touched block.
5. Optimizer overhead, including its own model calls.

If an optimizer reaches 5 percent of carried input and shortens that surface by
half, its raw ceiling is 2.5 percent before cache price, overhead or behavioral
effects. A 60 percent session-level claim is already dead. No paid run is
needed to learn that.

Then test wiring. Persist proof that the treatment fired on every treatment
trial. A null result means little if half the agent runs never invoked the hook.
Run a small smoke set to find broken commands and missing telemetry, though do
not estimate an effect from it. Coding-agent attempts are noisy enough that one
task can dominate an arm total.

The effect estimate comes from paired tasks under matched model, harness,
budget and environment. Compare per-task deltas rather than raw arm totals.
Use task-level outcome gates. Report uncertainty. Repeat the small set before
scaling, then run a workload broad enough to contain tasks the optimizer was
not designed around.

Most importantly, keep the token classes separate. A cache read token, a cold
input token and an output token occupy one context accounting system but carry
different prices. Collapsing them into a single raw total discards the economic
mechanism being tested.

## What a defensible result card contains

A public result should let a skeptical reader reconstruct the claim without
trusting the optimizer's dashboard. At minimum, publish:

- exact treatment and version;
- task corpus, exclusions and pairing rule;
- model, harness, effort, budget and repetition count;
- proof of treatment adoption;
- provider-reported usage by token class;
- per-task cost deltas and confidence interval;
- turns, tool calls and quality outcomes;
- failures and compatibility errors;
- declared primary endpoint chosen before the run.

Local reduction still belongs in the card because it explains mechanism and
helps debug a weak result. The paired bill remains the system-level evidence.

This discipline produces unglamorous findings. We watched one promising
rewrite save raw tokens while losing on cache-weighted cost. Elsewhere, a style
intervention worked perfectly but covered too little work; a tool claimed
millions of removed tokens as task cost rose.

Each result is useful. Failed savings claims show where the real cost lives:
inside repeated prefixes, cache boundaries and agent behavior. Counting fewer
bytes was the easy part.

## Sources

1. JetBrains, [Does speaking to agents like cavemen really save 65% of tokens? We test](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/), July 2026.
2. JetBrains, [Does `rtk` really cut agent tokens by 60–90%? We tested it](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/), July 2026.
3. Anthropic, [Claude API pricing: prompt caching](https://platform.claude.com/docs/en/about-claude/pricing), accessed 13 August 2026.
4. Xiao et al., [Reducing Cost of LLM Agents with Trajectory Reduction](https://arxiv.org/abs/2509.23586), FSE 2026.
5. Chen et al., [CoACT: Action-Preserving Observation Compression for Coding Agents](https://arxiv.org/abs/2607.02911), arXiv preprint, 2026.