Efficiency and the token economy

In 1865 the economist William Stanley Jevons noticed something that embarrassed the engineers of his day. Steam engines had become dramatically more efficient, and the expectation was that Britain would therefore burn less coal. Britain burned far more. Efficiency had made coal-powered work cheap enough to use in places nobody had considered using it, and the total consumption of the resource rose with every improvement in how little of it each machine needed.

The same thing has now happened to intelligence, faster than it happened to coal, electricity, bandwidth, or cloud compute. The price of a token has collapsed, and the amount companies spend on tokens has gone up. Almost every strategic mistake being made in this market right now traces back to reading that fact backwards.

I want to lay out what I think is actually happening, because I have spent fifteen months looking at very little else, and because the conclusions I have reached are not the ones most people in this industry are working from.

Why I ended up looking at this

I am 19. Last April I was building with coding agents and running out of money doing it, so I read my own session logs to find out where it had gone. Most of what I had paid for was not work. It was the model restating my question, recapping what it had just done, and re-sending the same context on every step of a loop that had no idea what it was spending.

I wrote a tool that fixed the small version of that problem. Then companies started asking me questions the tool could not answer, and I discovered that the small problem was a visible corner of a much larger one. This letter is about the larger one.

The price of a token is not the cost of AI

Blended inference costs fell roughly 67% year over year, from about $18.40 to $6.07 per million tokens between the first quarters of 2025 and 2026. Epoch AI's work on price trends puts the half-life of a fixed capability level at 1.1 to 1.55 years, faster than Moore's Law ever ran, with the decline rate varying from 9x to 900x per year depending on which performance milestone you hold constant. A frontier-class capability that cost $30 per million tokens three years ago now costs cents.

Over the same period, 73% of enterprises reported AI spend above their original projections. Per-developer AI consumption rose roughly 18.6x in nine months. Gartner published a forecast in June that AI coding token costs will overtake the average developer's salary by 2028. One large technology company exhausted its entire 2026 AI coding budget in April. Goldman Sachs projects global token consumption multiplying 24x by 2030 on the back of agent adoption.

Both sets of numbers are true at once, and the relationship between them is causal in the direction people find counterintuitive. Each price cut converts a category of work that was economically impossible into work that runs continuously. Cheap tokens do not reduce the bill. They are what creates it.

This has a direct consequence for anyone deciding where value accrues. The common argument against building anything in AI efficiency is that model providers will keep cutting prices until the problem disappears. That argument has now been tested against three years of price cuts and has been wrong every time. Falling unit prices are the mechanism by which consumption grows faster than prices fall. The efficiency question does not get smaller as tokens get cheaper. It gets larger, because the volume it applies to is compounding.

Agents changed what the bill is made of

A chat interface bills you for a conversation. An agent bills you for a loop.

In agentic coding workloads the ratio of input to output tokens runs about 20 to 25 to one. The same task through an agent consumes somewhere between 5 and 30 times the tokens of a single model call, and complex multi-agent pipelines run higher. The reason is structural: a transformer has no memory between calls, so every step of every loop re-sends the entire accumulated context. The system prompt, the tool schemas, the conversation, every prior tool result. All of it, every time, priced at full rate.

So most of an agent bill buys re-reading. The model pays full freight, on every step, to be shown things it was shown ninety seconds ago. That is the single most important economic fact in this market, because it determines whether there is anything to be done at all. If the bill were the price of thinking, the only lever would be to think less, and this would be a dull industry to work in. The bill is a property of how the system is assembled, which makes it addressable without giving up any capability.

The scale is no longer theoretical. One engineer reportedly consumed $81,267 of tokens in a single week on a side project. Developers on subscription plans routinely generate several thousand dollars of API-equivalent usage a month. Cost per completed task, which is the number that actually matters, currently lands somewhere between three and thirteen cents depending on the model and the tooling, and almost no organization can tell you their own figure.

Efficiency is not primarily a financial argument

This is where I part company with most people who work on AI cost.

Long context degrades model quality. Chroma's research tested 18 frontier models and found that every one of them performs worse as input length grows, and that the degradation arrives in cliffs, with models that handle 32,000 tokens cleanly collapsing at 64,000. Retrieval studies have shown accuracy falling from the low seventies to the high fifties once around twenty documents are in the window. Roughly 65% of enterprise AI failures have been attributed to context drift and memory loss during multi-step reasoning rather than to hitting a context limit. The model does not raise an error when this happens. It just gets quietly worse.

So the tokens that cost the most money are frequently the same tokens degrading the output. On a large share of real workloads, cutting them makes the agent better and cheaper in the same motion. Efficiency and capability point the same direction here, and the industry keeps filing them under different departments with different budgets and different meetings.

There is a second argument that will matter more than the financial one within a few years. Gartner projects that 40% of AI data centers will be power-constrained by 2027, and grid connection approvals in major US and European markets now run 24 to 36 months. When the binding constraint on AI is electricity rather than money, efficiency stops being a cost lever and becomes a capacity lever. The ratio of tokens to useful work determines how much AI a company, or a country, is able to run at all.

Those two facts are physically connected, which is a link I rarely see anyone make. At the input-to-output ratios agents produce, the prefill pass over the input dominates GPU time on a request. So a re-sent context occupies scarce silicon for as long as it takes to process. It is somebody's invoice and it is somebody's substation. A stable prefix that never gets cached is wasted electricity in a completely literal sense, and the industry's current answer to that is to build more power stations.

This connects to the argument being made publicly right now about diffusion. AI leadership will be decided by how deeply this technology spreads into ordinary work, with the right model matched to the right job at the right cost and frontier capability reserved for actual frontier problems. I agree with all of it. I would add the part that usually goes unsaid: diffusion is gated on cost per task, and cost per task is gated on efficiency. Efficiency is the mechanism that carries AI to the factory floor and to the small business that has never bought software in its life. It gets treated as a finance concern sitting downstream of the real engineering, and that is a filing error.

Every lever is a bet against quality

The optimization frontier is real and it is well understood by the people working on it. Provider caching is the largest immediate lever, with cache reads typically priced at a tenth of base input and cache writes at a 25% premium, so a stable prefix pays for itself after two hits. The lever is also fragile in ways that only show up in production: swap models mid-session and a 90%-off read becomes a 125%-of-base write across the entire accumulated context. Small models are the next lever, with a 7-billion-parameter model costing 10 to 30 times less to serve than a 70-to-175-billion one, and a well-routed 70/30 split running roughly ten times cheaper than sending everything to the frontier. Compression is the third. Rendering text into images is a fourth, and a striking one: DeepSeek's optical compression work holds 97% decoding precision at 10x compression and falls to about 60% at 20x.

Every one of those levers is a trade. Caching trades architectural rigidity. Small models trade capability on the hard cases. Compression trades detail, and it fails abruptly rather than gracefully: LLMLingua-2 degrades sharply as the compression rate rises, and past roughly 30% the agent tasks fail outright. Terminal output is the clearest example of why. The strings that decide whether an agent succeeds are the error text, the file path, the test name, the exact version number, and those are precisely the strings a generic summarizer discards as noise. There is now also a literature on compressed context as an attack surface, which is what happens when a security boundary and a cost boundary turn out to be the same boundary.

So the entire optimization frontier consists of removing things and hoping they did not matter. That is a perfectly reasonable engineering position. It is an indefensible accounting position, unless you can show what happened to quality when you removed them.

The industry cannot currently check its own work

Here is the part I find genuinely alarming, and I say that as someone whose business benefits from it being true.

The standard method for evaluating model output at scale is to have another model judge it. Large-scale studies published this year found frontier models exceeding 50% error rates on bias tests as judges, with consistency breaking down under changes as trivial as reformatting, paraphrasing, or a shift in verbosity. A dedicated harness for stress-testing judge reliability was released in March 2026 because the problem was serious enough to need one. Meanwhile the models themselves are retired within 12 to 18 months, so the baseline you measured against last year is frequently no longer purchasable. An analysis of 69 academic papers evaluating one provider's models found that 5 had code that would run and none reproduced their reported results.

Nobody in the value chain is positioned to referee

The measurement gap would close on its own if anyone in the market had an incentive to close it. Look at who would have to.

Model providers bill per token, so a token that accomplishes nothing is still revenue. Gateways take a percentage of volume, so a customer who spends less is worth less. Compression vendors publish their own benchmarks on their own sample slices, which buyers correctly discount. Observability platforms get paid whether the bill moves or not, and their provider integrations are deliberately read-only, which means they can describe the problem precisely and cannot touch it.

To be clear, none of this is bad faith. Every one of these companies is behaving rationally inside the position it occupies. That is exactly the problem. A market does not self-referee, and the referee cannot be the party earning a percentage of the thing being judged. Financial markets did not solve this with better intentions from brokers; they solved it with an accounting standard and an independent audit. Cloud computing went through a smaller version of the same process, and the FinOps community's FOCUS specification is now extending into token economics, with roughly 68% of organizations spending over $100 million a year already working from FOCUS-formatted data.

The token economy is roughly where cloud was in 2014, with one difference that changes what the standard has to do. This is the first major cost line in business where cutting it can silently degrade the product, and the degradation does not appear anywhere in the bill. Cloud spend never had that property. You could shut off an over-provisioned instance and the only thing that changed was the invoice. Here, the cheaper configuration and the worse configuration produce the same number in the accounting system and differ only in output quality that nobody is reliably measuring.

An accounting standard for this cannot only count dollars. It has to bind the dollar to evidence that the work held up. Nothing in the history of cost accounting has had to do that, which is why I think this ends up as a distinct layer of the stack instead of a feature bolted onto an existing one.

What follows from all this

The unit of account changes from the token to the task. Seat-based pricing fell from 21% to 15% of software companies in twelve months while per-resolution pricing went live at one to two dollars a conversation, and AI product gross margins are averaging around 52% against the 80% that defined mature software, with inference consuming roughly 23% of revenue at scaling AI companies. Public companies began disclosing inference cost ratios separately in the first quarter of this year. When you sell outcomes, tokens are your cost of goods sold, and no serious business sells at a margin it cannot measure. Cost per completed task becomes a reported figure with a definition someone can audit, and I would expect that to happen within about three years.

The default agent stops being one large model and becomes a heterogeneous system that routes each step to the cheapest capable option. This is already the technical consensus. What is underappreciated is the ordering: you cannot route downward safely without a way to prove the cheap path held quality, which makes verification a prerequisite for routing rather than a nice addition to it. Most people in this market have that dependency backwards, and it is the main reason automatic routing has not become the default already.

It also has to be automatic. One of the more deflating figures in this whole market comes from published gateway data: cut a model's price by 10% and usage shifts by well under 1%. Buyers do not chase cheap tokens. They have work to do, and the cognitive cost of choosing a model per task exceeds the money saved by choosing correctly. Any theory of AI efficiency that depends on developers making good procurement decisions inside a loop running thousands of times an hour is not a theory, it is a hope. The saving happens when a system makes the choice and takes responsibility for the quality of the result.

Context becomes managed state instead of a text buffer. Memory, caching, compaction, and eventually encodings that are not text at all. Whoever manages that state ends up controlling the bill and the output quality with the same set of decisions, which is why I do not think these end up as two separate products.

And the thing that becomes genuinely scarce is proof. Everything above is a trade made against quality at machine speed, and a trade you cannot verify is a guess you are invoicing for.

What I am building

The sentence I have settled on is that Caveman is the operating layer that makes every AI agent think, remember and execute more efficiently.

That is a jargon sentence and I am aware of it, so here is the plain version. Agents waste money in three places, and there is a fourth thing that makes fixing any of them safe.

They waste it on thinking out loud. An agent generates explanation, restatement and hedging that no human will ever read, because the thing receiving it is a program. That gets cut, which is where this project started.

They waste it on forgetting. An agent re-sends its entire context every step and re-derives conclusions it reached an hour ago, because the model has no memory between calls. That gets held onto and handed back instead of bought again.

They waste it on overkill. An agent sends a request to rename a variable to the same model it uses to design a system, because someone picked one model in a config file six months ago and nothing has re-examined that decision since. That gets routed to the cheapest model that still does the job.

Each of those three is a bet against quality, which is the entire argument of this letter, so the fourth part is the one that matters: it runs the customer's own checks against their own traffic before and after, and counts only the savings that survived. Cheaper is easy. Cheaper and provably intact is the product.

That last part is the whole design, and it is the reason the architecture looks the way it does. Four money figures exist in the system and never mix: what the traffic cost at published list prices, what the telemetry suggests is available, what a before-and-after comparison indicates, and what has been causally proven on live traffic. Only the last one is permitted to be called a saving. Everything else is labeled as the estimate it is.

Holding that line is expensive. It means our headline number is smaller than the one on a competitor's landing page, and it means occasionally telling a customer that the improvement we shipped them counts as zero until the evidence catches up with it. I do it because the number is the entire product. A referee that rounds in its own favor once is not a referee afterwards, and the person buying this has to defend the figure to someone upstairs who will check it.

The objection worth taking seriously

To be fair to the other side of this, the strongest counterargument concedes the whole problem and says it gets solved generically. Model providers ship trustworthy per-request quality signals themselves, buyers get verification free from the same invoice that bills them, and a separate layer becomes redundant.

That outcome would be good for the world and I would take it. I think it is unlikely for a structural reason rather than a technical one. It requires the party selling tokens to certify the occasions when fewer tokens would have done the job. Every industry that eventually needed that kind of certainty has ended up getting it from someone standing outside the transaction, and it has usually arrived because buyers demanded it rather than because sellers offered it.

Why this matters beyond one company

The prize here is bigger than a category of software. Intelligence is becoming the primary variable input to most knowledge work, and for the first time in computing that input has a price, a supply constraint, and a quality that varies with how cheaply you buy it. Every organization that adopts AI seriously will end up asking the same three questions: what did this cost, what did it produce, and would a cheaper configuration have produced the same thing. Right now essentially nobody can answer the third with evidence.

If that stays true, AI diffuses more slowly than it should, because finance departments do not fund what they cannot measure and 25% of planned AI spend has already been pushed into next year on exactly those grounds. If it becomes answerable, the cost of an AI-performed task falls to what it should actually be, and the technology reaches the enormous number of use cases that are viable at three cents and not at thirty.

I am 19, I found this by reading my own logs because I could not afford my own hobby, and I have been wrong about parts of it repeatedly. The version of this letter I would have written in April was about making prompts smaller, which I now think is roughly 10% of the story. What I am confident about is the shape: work is moving to machines, machines bill by the token, the bill is mostly waste rather than thought, and every fix for that is a bet against quality that somebody has to be able to settle.

That layer is going to exist. I would like to be the one who builds it. If you think I have any of this wrong, that is a considerably more useful reply than agreement.

Julius