How removing tokens raised the bill

In July, a tool designed to shrink coding-agent output reported that it had saved 96.2 million tokens. The paired provider bill moved in the other direction. Across 80 clean task pairs at low reasoning effort, the treatment cost 7.6 percent more. It also produced 13.8 percent more turns and 14.3 percent more cache reads.

Both measurements were correct within their own frames: one counted characters removed when a shell command returned, while the provider counted what it charged to complete the task. Between those counters sits the agent loop, where a small edit to one observation can change future actions, invalidate a prompt cache, or provoke another tool call.

That is the trap in agent-efficiency work. Local tokens are easy to count; cost belongs to the entire trajectory.

A block is charged for its future

A chat request has one input and one output. A coding agent has a growing prefix. Each turn sends the system instructions, tool definitions, conversation and earlier tool results back to the model. Prompt caching may discount the stable part, though those tokens still occupy context and still affect what the model sees.

For a session with T model calls, a useful first approximation is:

input work = sum(prefix at turn t), for t = 1..T

A 4,000-token test log inserted after turn three of a 30-turn session can be carried through roughly 27 later calls. Counted once, it is 4,000 tokens. Counted as context processed over the remaining trajectory, it is about 108,000 token-occurrences before cache pricing.

The same multiplication applies to fixed instructions and tool schemas, making turn count unusually dangerous. Each extra turn pays for the accumulated prefix alongside its new response.

That gives any token-reduction claim three separate denominators:

The first counter is useful engineering telemetry. It is a weak savings claim.

Prompt caching changes the sign

Current Claude pricing makes the cache arithmetic easy to see: a five-minute cache write costs 1.25 times the base input rate, versus 0.1 times for a cache hit. One read repays the write premium; later reads make a stable prefix cheaper still.

Now suppose an optimizer revises an old 10,000-token block after it has entered that stable prefix. The revised block may be shorter, but the edit breaks exact prefix identity at that point. The next request must ingest the changed suffix again. A token removed from the old block saves a future read at 0.1 times the base rate; the rewrite can trigger fresh ingestion at 1 or 1.25 times base.

The relevant comparison is therefore weighted:

net effect
  = discounted future reads avoided
  - new cache write or cold-input work
  - optimizer call cost
  - cost of any additional agent turns

Raw token count is only one term. Expected remaining calls determine whether timing and prefix location turn a rewrite into a win.

We saw the sign flip in a fixed-trajectory replay over 24 recorded coding sessions. In one 12-session arm, an aged-context rewrite reduced modeled raw tokens by 0.75 percent while raising cache-weighted cost by 0.83 percent. In the other arm, raw tokens fell 0.44 percent and weighted cost rose 2.08 percent. The culprit was re-ingestion: the rewrite saved bytes and rebuilt cached suffixes.

Those figures are an inferred mechanism study, not a measured savings result. The replay held agent behavior fixed, reconstructed 80.2 percent of provider-reported tokens in one arm and 35.1 percent in the other, and could not observe the extra turns a lossy edit might cause. Its value is narrower. It proves that cache arithmetic alone can reverse a token win before behavior even enters the calculation.

The model can spend the bytes again

Behavior creates the larger failure mode. A shortened terminal result may omit the exact test name, path, version or error line that determines the next move. The agent rarely announces the loss. It searches again, reruns the test, reads the file, or takes another plausible route. Every recovery action adds a turn and carries the prefix once more.

JetBrains' paired evaluation of rtk, the tool behind the 96.2-million counter, found a related system-level penalty without isolating this recovery mechanism. Its transcript review found only a few compression-induced rereads and no single culprit; ordinary trajectory variance explained much of the movement. The hook also reached only a subset of shell results. Built-in file reads bypassed it, many commands sat outside its rules, and Claude Code already truncated extreme outputs. A free transcript replay put the likely input-token ceiling near 3 percent. Paid trials then found a higher bill, with greater rewrite exposure correlated with a larger penalty at low effort. Correlation did not identify the cause.

The tool's local analytics still reported a 99.8 percent reduction across the content it touched. One cat of a 1.2 MB CSV received credit for roughly 320,000 saved tokens even though the harness would have truncated the raw result before sending it to the model. The counterfactual existed only inside the tool's accounting.

We learned a related lesson from an independent evaluation of our own Caveman writing skill. Short chat-style answers had suggested a 65 percent output-token reduction. JetBrains forced the skill on across 82 clean paired coding tasks and measured 8.5 percent. Code, diffs, tool calls and error strings dominated the agent's output, and the skill correctly left them alone. Quality was statistically indistinguishable. The original percentage described the slice we had compressed, then escaped its denominator and became a claim about the whole workload.

That is how honest component measurements turn into bad system claims. Nothing needs to be fabricated. A denominator merely goes missing.

Run the cheap experiment first

Before paying for a large paired benchmark, replay existing transcripts and bound the surface an optimizer can reach. This does not establish savings. It can kill impossible claims at almost no cost.

Start with five quantities:

1. Total provider-reported input, split into uncached input, cache creation and cache reads.
2. Output and reasoning usage.
3. Share of the trajectory the optimizer can touch after harness truncation and tool exclusions.
4. Carry depth: how many later model calls receive each touched block.
5. Optimizer overhead, including its own model calls.

If an optimizer reaches 5 percent of carried input and shortens that surface by half, its raw ceiling is 2.5 percent before cache price, overhead or behavioral effects. A 60 percent session-level claim is already dead. No paid run is needed to learn that.

Then test wiring. Persist proof that the treatment fired on every treatment trial. A null result means little if half the agent runs never invoked the hook. Run a small smoke set to find broken commands and missing telemetry, though do not estimate an effect from it. Coding-agent attempts are noisy enough that one task can dominate an arm total.

The effect estimate comes from paired tasks under matched model, harness, budget and environment. Compare per-task deltas rather than raw arm totals. Use task-level outcome gates. Report uncertainty. Repeat the small set before scaling, then run a workload broad enough to contain tasks the optimizer was not designed around.

Most importantly, keep the token classes separate. A cache read token, a cold input token and an output token occupy one context accounting system but carry different prices. Collapsing them into a single raw total discards the economic mechanism being tested.

What a defensible result card contains

A public result should let a skeptical reader reconstruct the claim without trusting the optimizer's dashboard. At minimum, publish:

• exact treatment and version;
• task corpus, exclusions and pairing rule;
• model, harness, effort, budget and repetition count;
• proof of treatment adoption;
• provider-reported usage by token class;
• per-task cost deltas and confidence interval;
• turns, tool calls and quality outcomes;
• failures and compatibility errors;
• declared primary endpoint chosen before the run.

Local reduction still belongs in the card because it explains mechanism and helps debug a weak result. The paired bill remains the system-level evidence.

This discipline produces unglamorous findings. We watched one promising rewrite save raw tokens while losing on cache-weighted cost. Elsewhere, a style intervention worked perfectly but covered too little work; a tool claimed millions of removed tokens as task cost rose.

Each result is useful. Failed savings claims show where the real cost lives: inside repeated prefixes, cache boundaries and agent behavior. Counting fewer bytes was the easy part.

Sources

1. JetBrains, Does speaking to agents like cavemen really save 65% of tokens? We test, July 2026.
2. JetBrains, Does rtk really cut agent tokens by 60–90%? We tested it, July 2026.
3. Anthropic, Claude API pricing: prompt caching, accessed 13 August 2026.
4. Xiao et al., Reducing Cost of LLM Agents with Trajectory Reduction, FSE 2026.
5. Chen et al., CoACT: Action-Preserving Observation Compression for Coding Agents, arXiv preprint, 2026.