---
title: "Efficiency and the token economy"
description: "In 1865 the economist William Stanley Jevons noticed something that embarrassed"
canonical: https://caveman.so/news/efficiency-and-the-token-economy
last-updated: 2026-07-26
---

# Efficiency and the token economy

In 1865 the economist William Stanley Jevons noticed something that embarrassed
the engineers of his day. Steam engines had become dramatically more efficient,
and the expectation was that Britain would therefore burn less coal. Britain
burned far more. Efficiency had made coal-powered work cheap enough to use in
places nobody had considered using it, and the total consumption of the resource
rose with every improvement in how little of it each machine needed.

The same thing has now happened to intelligence, faster than it happened to coal,
electricity, bandwidth, or cloud compute. The price of a token has collapsed, and
the amount companies spend on tokens has gone up. Almost every strategic mistake
being made in this market right now traces back to reading that fact backwards.

I want to lay out what I think is actually happening, because I have spent
fifteen months looking at very little else, and because the conclusions I have
reached are not the ones most people in this industry are working from.

## Why I ended up looking at this

I am 19. Last April I was building with coding agents and running out of money
doing it, so I read my own session logs to find out where it had gone. Most of
what I had paid for was not work. It was the model restating my question,
recapping what it had just done, and re-sending the same context on every step of
a loop that had no idea what it was spending.

I wrote a tool that fixed the small version of that problem. Then companies
started asking me questions the tool could not answer, and I discovered that the
small problem was a visible corner of a much larger one. This letter is about the
larger one.

## The price of a token is not the cost of AI

Blended inference costs fell roughly 67% year over year, from about $18.40 to
$6.07 per million tokens between the first quarters of 2025 and 2026. Epoch AI's
work on price trends puts the half-life of a fixed capability level at 1.1 to
1.55 years, faster than Moore's Law ever ran, with the decline rate varying from
9x to 900x per year depending on which performance milestone you hold constant.
A frontier-class capability that cost $30 per million tokens three years ago now
costs cents.

Over the same period, 73% of enterprises reported AI spend above their original
projections. Per-developer AI consumption rose roughly 18.6x in nine months.
Gartner published a forecast in June that AI coding token costs will overtake the
average developer's salary by 2028. One large technology company exhausted its
entire 2026 AI coding budget in April. Goldman Sachs projects global token
consumption multiplying 24x by 2030 on the back of agent adoption.

Both sets of numbers are true at once, and the relationship between them is
causal in the direction people find counterintuitive. Each price cut converts a
category of work that was economically impossible into work that runs
continuously. Cheap tokens do not reduce the bill. They are what creates it.

This has a direct consequence for anyone deciding where value accrues. The common
argument against building anything in AI efficiency is that model providers will
keep cutting prices until the problem disappears. That argument has now been
tested against three years of price cuts and has been wrong every time. Falling
unit prices are the mechanism by which consumption grows faster than prices fall.
The efficiency question does not get smaller as tokens get cheaper. It gets
larger, because the volume it applies to is compounding.

## Agents changed what the bill is made of

A chat interface bills you for a conversation. An agent bills you for a loop.

In agentic coding workloads the ratio of input to output tokens runs about 20 to
25 to one. The same task through an agent consumes somewhere between 5 and 30
times the tokens of a single model call, and complex multi-agent pipelines run
higher. The reason is structural: a transformer has no memory between calls, so
every step of every loop re-sends the entire accumulated context. The system
prompt, the tool schemas, the conversation, every prior tool result. All of it,
every time, priced at full rate.

So most of an agent bill buys re-reading. The model pays full freight, on every
step, to be shown things it was shown ninety seconds ago. That is the single most
important economic fact in this market, because it determines whether there is
anything to be done at all. If the bill were the price of thinking, the only lever
would be to think less, and this would be a dull industry to work in. The bill is
a property of how the system is assembled, which makes it addressable without
giving up any capability.

The scale is no longer theoretical. One engineer reportedly consumed $81,267 of
tokens in a single week on a side project. Developers on subscription plans
routinely generate several thousand dollars of API-equivalent usage a month. Cost
per completed task, which is the number that actually matters, currently lands
somewhere between three and thirteen cents depending on the model and the tooling,
and almost no organization can tell you their own figure.

## Efficiency is not primarily a financial argument

This is where I part company with most people who work on AI cost.

Long context degrades model quality. Chroma's research tested 18 frontier models
and found that every one of them performs worse as input length grows, and that
the degradation arrives in cliffs, with models that handle 32,000 tokens cleanly
collapsing at 64,000. Retrieval studies have shown accuracy falling from the low
seventies to the high fifties once around twenty documents are in the window.
Roughly 65% of
enterprise AI failures have been attributed to context drift and memory loss
during multi-step reasoning rather than to hitting a context limit. The model does
not raise an error when this happens. It just gets quietly worse.

So the tokens that cost the most money are frequently the same tokens degrading
the output. On a large share of real workloads, cutting them makes the agent
better and cheaper in the same motion. Efficiency and capability point the same
direction here, and the industry keeps filing them under different departments
with different budgets and different meetings.

There is a second argument that will matter more than the financial one within a
few years. Gartner projects that 40% of AI data centers will be power-constrained
by 2027, and grid connection approvals in major US and European markets now run 24
to 36 months. When the binding constraint on AI is electricity rather than money,
efficiency stops being a cost lever and becomes a capacity lever. The ratio of
tokens to useful work determines how much AI a company, or a country, is able to
run at all.

Those two facts are physically connected, which is a link I rarely see anyone
make. At the input-to-output ratios agents produce, the prefill pass over the
input dominates GPU time on a request. So a re-sent context occupies scarce
silicon for as long as it takes to process. It is somebody's invoice and it is
somebody's substation. A stable prefix that never gets cached is wasted
electricity in a completely literal sense, and the industry's current answer to
that is to build more power stations.

This connects to the argument being made publicly right now about diffusion. AI
leadership will be decided by how deeply this technology spreads into ordinary
work, with the right model matched to the right job at the right cost and frontier
capability reserved for actual frontier problems. I agree with all of it. I would
add the part that usually goes unsaid: diffusion is gated on cost per task, and
cost per task is gated on efficiency. Efficiency is the mechanism that carries AI
to the factory floor and to the small business that has never bought software in
its life. It gets treated as a finance concern sitting downstream of the real
engineering, and that is a filing error.

## Every lever is a bet against quality

The optimization frontier is real and it is well understood by the people working
on it. Provider caching is the largest immediate lever, with cache reads typically
priced at a tenth of base input and cache writes at a 25% premium, so a stable
prefix pays for itself after two hits. The lever is also fragile in ways that only
show up in production: swap models mid-session and a 90%-off read becomes a
125%-of-base write across the entire accumulated context. Small models are the
next lever, with a 7-billion-parameter model costing 10 to 30 times less to serve
than a 70-to-175-billion one, and a well-routed 70/30 split running roughly ten
times cheaper than sending everything to the frontier. Compression is the third.
Rendering text into images is a fourth, and a striking one: DeepSeek's optical
compression work holds 97% decoding precision at 10x compression and falls to
about 60% at 20x.

Every one of those levers is a trade. Caching trades architectural rigidity.
Small models trade capability on the hard cases. Compression trades detail, and
it fails abruptly rather than gracefully: LLMLingua-2 degrades sharply as the
compression rate rises, and past roughly 30% the agent tasks fail outright.
Terminal output is the clearest example of why. The strings that decide whether an
agent succeeds are the error text, the file path, the test name, the exact
version number, and those are precisely the strings a generic summarizer discards
as noise. There is now also a literature on compressed context as an attack
surface, which is what happens when a security boundary and a cost boundary turn
out to be the same boundary.

So the entire optimization frontier consists of removing things and hoping they
did not matter. That is a perfectly reasonable engineering position. It is an
indefensible accounting position, unless you can show what happened to quality
when you removed them.

## The industry cannot currently check its own work

Here is the part I find genuinely alarming, and I say that as someone whose
business benefits from it being true.

The standard method for evaluating model output at scale is to have another model
judge it. Large-scale studies published this year found frontier models exceeding
50% error rates on bias tests as judges, with consistency breaking down under
changes as trivial as reformatting, paraphrasing, or a shift in verbosity. A
dedicated harness for stress-testing judge reliability was released in March 2026
because the problem was serious enough to need one. Meanwhile the models
themselves are retired within 12 to 18 months, so the baseline you measured
against last year is frequently no longer purchasable. An analysis of 69 academic
papers evaluating one provider's models found that 5 had code that would run and
none reproduced their reported results.

So this industry is about to make millions of quality-for-cost trades per second,
automatically, at scale, without a dependable way to check whether any single one
of them was a good trade. That is the actual state of the art in July 2026.

## Nobody in the value chain is positioned to referee

The measurement gap would close on its own if anyone in the market had an
incentive to close it. Look at who would have to.

Model providers bill per token, so a token that accomplishes nothing is still
revenue. Gateways take a percentage of volume, so a customer who spends less is
worth less. Compression vendors publish their own benchmarks on their own sample
slices, which buyers correctly discount. Observability platforms get paid whether
the bill moves or not, and their provider integrations are deliberately read-only,
which means they can describe the problem precisely and cannot touch it.

To be clear, none of this is bad faith. Every one of these companies is behaving
rationally inside the position it occupies. That is exactly the problem. A market
does not self-referee, and the referee cannot be the party earning a percentage
of the thing being judged. Financial markets did not solve this with better
intentions from brokers; they solved it with an accounting standard and an
independent audit. Cloud computing went through a smaller version of the same
process, and the FinOps community's FOCUS specification is now extending into
token economics, with roughly 68% of organizations spending over $100 million a
year already working from FOCUS-formatted data.

The token economy is roughly where cloud was in 2014, with one difference that
changes what the standard has to do. **This is the first major cost line in
business where cutting it can silently degrade the product, and the degradation
does not appear anywhere in the bill.** Cloud spend never had that
property. You could shut off an over-provisioned instance and the only thing that
changed was the invoice. Here, the cheaper configuration and the worse
configuration produce the same number in the accounting system and differ only in
output quality that nobody is reliably measuring.

An accounting standard for this cannot only count dollars. It has to bind the
dollar to evidence that the work held up. Nothing in the history of cost
accounting has had to do that, which is why I think this ends up as a distinct
layer of the stack instead of a feature bolted onto an existing one.

## What follows from all this

The unit of account changes from the token to the task. Seat-based pricing fell
from 21% to 15% of software companies in twelve months while per-resolution
pricing went live at one to two dollars a conversation, and AI product gross
margins are averaging around 52% against the 80% that defined mature software,
with inference consuming roughly 23% of revenue at scaling AI companies. Public
companies began disclosing inference cost ratios separately in the first quarter
of this year. When you sell outcomes, tokens are your cost of goods sold, and no
serious business sells at a margin it cannot measure. Cost per completed task
becomes a reported figure with a definition someone can audit, and I would expect
that to happen within about three years.

The default agent stops being one large model and becomes a heterogeneous system
that routes each step to the cheapest capable option. This is already the
technical consensus. What is underappreciated is the ordering: you cannot route
downward safely without a way to prove the cheap path held quality, which makes
verification a prerequisite for routing rather than a nice addition to it. Most
people in this market have that dependency backwards, and it is the main reason
automatic routing has not become the default already.

It also has to be automatic. One of the more deflating figures in this whole
market comes from published gateway data: cut a model's price by 10% and usage
shifts by well under 1%. Buyers do not chase cheap tokens. They have work to do,
and the cognitive cost of choosing a model per task exceeds the money saved by
choosing correctly. Any theory of AI efficiency that depends on developers making
good procurement decisions inside a loop running thousands of times an hour is
not a theory, it is a hope. The saving happens when a system makes the choice and
takes responsibility for the quality of the result.

Context becomes managed state instead of a text buffer. Memory, caching,
compaction, and eventually encodings that are not text at all. Whoever manages
that state ends up controlling the bill and the output quality with the same set
of decisions, which is why I do not think these end up as two separate products.

And the thing that becomes genuinely scarce is proof. Everything above is a trade
made against quality at machine speed, and a trade you cannot verify is a guess
you are invoicing for.

## What I am building

Caveman makes every AI agent think, remember and execute on a fraction of the
cost — and proves it.

That is a jargon sentence and I am aware of it, so here is the plain version.
Agents waste money in three places, and there is a fourth thing that makes fixing
any of them safe.

They waste it on thinking out loud. An agent generates explanation, restatement
and hedging that no human will ever read, because the thing receiving it is a
program. That gets cut, which is where this project started.

They waste it on forgetting. An agent re-sends its entire context every step and
re-derives conclusions it reached an hour ago, because the model has no memory
between calls. That gets held onto and handed back instead of bought again.

They waste it on overkill. An agent sends a request to rename a variable to the
same model it uses to design a system, because someone picked one model in a
config file six months ago and nothing has re-examined that decision since. That
gets routed to the cheapest model that still does the job.

Each of those three is a bet against quality, which is the entire argument of this
letter, so the fourth part is the one that matters: it runs the customer's own
checks against their own traffic before and after, and counts only the savings
that survived. Cheaper is easy. Cheaper and provably intact is the product.

That last part is the whole design, and it is the reason the architecture looks
the way it does. Four money figures exist in the system and never mix: what the
traffic cost at published list prices, what the telemetry suggests is available,
what a before-and-after comparison indicates, and what has been causally proven on
live traffic. Only the last one is permitted to be called a saving. Everything
else is labeled as the estimate it is.

Holding that line is expensive. It means our headline number is smaller than the
one on a competitor's landing page, and it means occasionally telling a customer
that the improvement we shipped them counts as zero until the evidence catches up
with it. I do it because the number is the entire product. A referee that rounds
in its own favor once is not a referee afterwards, and the person buying this has
to defend the figure to someone upstairs who will check it.

## The objection worth taking seriously

To be fair to the other side of this, the strongest counterargument concedes the
whole problem and says it gets solved generically. Model providers ship
trustworthy per-request quality signals themselves, buyers get verification free
from the same invoice that bills them, and a separate layer becomes redundant.

That outcome would be good for the world and I would take it. I think it is
unlikely for a structural reason rather than a technical one. It requires the
party selling tokens to certify the occasions when fewer tokens would have done
the job. Every industry that eventually needed that kind of certainty has ended up
getting it from someone standing outside the transaction, and it has usually
arrived because buyers demanded it rather than because sellers offered it.

## Why this matters beyond one company

The prize here is bigger than a category of software. Intelligence is becoming the
primary variable input to most knowledge work, and for the first time in computing
that input has a price, a supply constraint, and a quality that varies with how
cheaply you buy it. Every organization that adopts AI seriously will end
up asking the same three questions: what did this cost, what did it produce, and
would a cheaper configuration have produced the same thing. Right now essentially
nobody can answer the third with evidence.

If that stays true, AI diffuses more slowly than it should, because finance
departments do not fund what they cannot measure and 25% of planned AI spend has
already been pushed into next year on exactly those grounds. If it becomes
answerable, the cost of an AI-performed task falls to what it should actually be,
and the technology reaches the enormous number of use cases that are viable at
three cents and not at thirty.

I am 19, I found this by reading my own logs because I could not afford my own
hobby, and I have been wrong about parts of it repeatedly. The version of this
letter I would have written in April was about making prompts smaller, which I now
think is roughly 10% of the story. What I am confident about is the shape: work is
moving to machines, machines bill by the token, the bill is mostly waste rather
than thought, and every fix for that is a bet against quality that somebody has to
be able to settle.

That layer is going to exist. I would like to be the one who builds it. If you
think I have any of this wrong, that is a considerably more useful reply than
agreement.

Julius