A benchmark printed 26.5%. We refused the result

The most attractive number printed by our latest agent-efficiency benchmark was 26.48%. Across the task pairs that held quality, its aggregate arithmetic put provider-reported tokens 26.48% lower for the candidate than for the baseline.

Every figure below is diagnostic evidence from an unpublishable run, not a benchmark result or savings claim.

That number will stay out of a savings headline.

The reason sits beside it in the report: only nine pairs held quality, and the paired bootstrap interval crossed zero. Aggregate output tokens rose 37.18%, the candidate verifier passed 75% of the full set, and every task ran once. The report marked itself publishable:false. Its blocker list counted the quality-held cluster floor and pair floor separately; the zero-crossing interval was the third.

This is the part of efficiency research that usually disappears between the experiment and the launch post. A diagnostic point estimate looks like an answer. The rest of the table shows how easily that answer can move, what it traded away, and whether the measurement supports the words a reader will attach to it.

The diagnostic looked ready for a headline

The experiment contained 12 paired, source-generated, tool-output-heavy investigation tasks over paginated API JSON, logs, CSV ledgers, and configuration dumps. Each task ran through a baseline arm and a candidate arm under the same benchmark contract. Pairing matters because agent workloads differ enormously. Comparing one difficult investigation in the candidate arm with an easy lookup in the baseline would mostly measure task selection. A pair compares two treatments on the same job.

Nine pairs passed the quality-held gate and entered the reported efficiency comparison. On those pairs, the artifact printed an aggregate total-token reduction of 26.48%, using a convention where reduction is positive:

reduction = (sum of baseline tokens - sum of candidate tokens) / sum of baseline tokens

For the interval discussion below, we also use signed change, which equals the negative of reduction. Under that convention the observed change was -26.48%.

The total metric combined provider-reported input with output, cache-read, and cache-write tokens without applying price weights; reasoning remained separate because providers do not all expose it the same way. Narrow scope. This metric answers a question about raw volume, leaving run cost and behavior on another sample unresolved.

Even with those boundaries, the 26.48% diagnostic tells us the candidate is worth another experiment. It also explains why teams get into trouble here. Remove the sample size and interval, round the number to 26%, and an exploratory result turns into a product claim with almost no visible effort.

The interval reversed the publication decision

The paired bootstrap resampled task identities with replacement while preserving the baseline-candidate relationship inside each task. Each resample produced a new aggregate effect, building a distribution of results that could arise from the observed task set.

Under the artifact's positive-reduction convention, the stored 95% interval ran from -12.47% to +53.58%. Its lower end represents a token increase; its upper end represents a large reduction. Expressing the same arithmetic as signed change reverses every sign: the diagnostic point becomes -26.48%, with an interval from -53.58% to +12.47%. Both forms describe the same uncertainty. Publishing the signs without defining the metric is an easy way to make a bad result look favorable.

Because zero sits inside either interval, the data remain compatible with no reduction. A meaningful increase also fits. The diagnostic point estimate is the center of the story people want to tell; the interval contains several stories the sample cannot yet rule out.

Nine clusters make that fragility unsurprising, since one unusual task can move the aggregate and a bootstrap cannot manufacture diversity absent from the original sample. Ten thousand resamples still contain information from nine tasks; computation reveals the small experiment's uncertainty without curing it.

The benchmark had a predeclared publication floor requiring at least ten quality-held clusters and pairs. It reached nine. That one-pair shortfall is easy to dismiss after seeing a promising estimate, which is exactly why the rule must exist before results arrive. A gate invented afterward is editorial judgment wearing a statistics badge.

Quality decides whether efficiency counts

Token reduction has no independent value when the agent stops completing the task. That sounds obvious until an experiment produces a dramatic usage chart. Then teams begin treating quality as a secondary panel, or they average a small regression into a large token win.

This report handled quality first. A pair had to hold the benchmark's required outcome before its token delta could enter the diagnostic efficiency comparison. Three of the 12 candidate runs failed the verifier, leaving a candidate verifier pass rate of 75%. Only nine pairs remained in the quality-held comparison.

The verifier itself also needs evidence. A deterministic check may prove that a file exists or JSON conforms to a schema while missing whether the agent solved the user's problem. An LLM judge can catch semantic failures but introduces model bias and grader drift; human review reaches subtleties the automated check misses at a cost that limits coverage. The right verifier depends on the task, and its false passes are part of the experiment's risk.

The Intervention Paradox study makes the distinction sharp by testing a critic with an offline AUROC near 0.94. Deployment told a different story. On one agent, interventions caused a 26-percentage-point success collapse because the action triggered by a correct warning could disrupt a trajectory that would have succeeded. Verifier accuracy and intervention safety are separate measurements.

For an optimizer, the equivalent failure is correctly flagging apparent waste and then damaging the answer path when changing it, which is why an outcome gate must test the changed system on the same task. A detector score or a plausible mechanism cannot substitute for that pair.

Total tokens hid an output trade

The aggregate token arithmetic moved down while candidate output tokens rose 37.18%. That trade changes what the total means.

An agent can consume less repeated input and generate a longer answer, or reduce cache writes while adding repair text; a useful transformation may spend more output to avoid a much larger input cost. Raw total tokens collapse those behaviors into one unit even though providers price input, output, cache reads, and cache creation differently.

Suppose a candidate saves 100,000 cache-read tokens and adds 10,000 output tokens. Raw volume improves by 90,000, yet cost may improve by far less when cache reads are cheap and output is expensive; changing the provider or model can alter the result again. Catalog-weighted cost must use the applicable model prices and token buckets. This experiment measured raw tokens, so it supports no cost-savings percentage.

The output increase also deserves a product reading. Longer output might carry needed detail. It might be verbosity that another agent will ingest on the next turn, turning today's output into tomorrow's input. The benchmark needs a quality check sensitive to completeness and an efficiency report that keeps token buckets separate. One total cannot settle both questions.

One repetition measures a run

Each arm ran once per task. Agent systems are stochastic, and one repetition cannot show how often a result survives sampling variation, tool timing, or a different valid execution path. Pairing controls task identity. It leaves within-task run-to-run variance largely unmeasured.

More repetitions alone would still leave a design problem if the protocol changed after inspecting the first result. Before the next run, freeze the primary metric and quality criteria in a protocol that also names exclusion handling. Define the bootstrap unit explicitly. The publication rule must already be locked. Record the workload version too, along with the model configuration and price catalog when money enters the claim.

Preregistration protects against a common form of accidental overfitting. A team runs several metrics, notices which one improved, then writes the experiment as if that metric had always been primary. Removing failing tasks after seeing their deltas requires no fraud; flexible analysis naturally selects the most flattering path through noisy data.

Fresh tasks matter as much as repeated runs. A candidate tuned against this small workload can learn its quirks. The next stage needs a locked holdout with enough quality-held pairs to narrow the interval, plus repeated executions to measure stability. Workload slices should remain visible; a healthy aggregate can conceal a failure concentrated in one task class.

What would earn the claim

This artifact sits on the first useful rung: an offline paired signal with provider-reported token counts. It identifies a candidate and records negative evidence alongside the favorable diagnostic point. That is research progress. It is still several steps away from a public savings percentage.

The next rung is a preregistered paired replication on fresh tasks, with more than one execution per arm and a validated outcome gate; a stable result then needs catalog-weighted cost that includes the output increase and every failed or excluded run under predeclared rules. After deployment, a controlled holdout can test the change on representative traffic while monitoring quality, then answer whether the system caused a durable saving in its actual environment.

This sequence matches the evaluation loop documented across current agent tooling: build versioned datasets from real failures, compare versions offline on the same inputs, then monitor the deployed version. Offline evaluation finds regressions before users do. Online evaluation catches distribution shifts the benchmark never contained. Mixing the two stages lets a promising lab number inherit production authority it has not earned.

Read the unpublishable diagnostic precisely: nine quality-held pairs yielded aggregate total-token reduction arithmetic of 26.48% under the artifact's convention. Its stored reduction interval was -12.47% to +53.58%; aggregate output rose 37.18%, the candidate verifier passed 75%, and there was one repetition. Enough for a rerun. We did not earn a benchmark result or savings claim.

Sources

• CaveBench paired aggregate, generated 8 August 2026: 12 source-generated, tool-output-heavy investigation tasks over paginated API JSON, logs, CSV ledgers, and configuration dumps; one repetition and nine quality-held pairs. Public reporting here is limited to tracked aggregate results; prompts and private traces are excluded.
• Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
• LangSmith, Evaluation concepts
• Braintrust, Evaluate systematically
• Phoenix, Datasets and experiments
• OpenAI, Graders API reference