---
title: "The average stayed flat. 125 tasks changed outcome."
description: "An agent optimization was evaluated across 998 matched software-engineering"
canonical: https://caveman.so/news/the-average-stayed-flat
last-updated: 2026-08-13
---

# The average stayed flat. 125 tasks changed outcome.

An agent optimization was evaluated across 998 matched software-engineering
tasks. It recovered 66 tasks the baseline had failed and broke 59 tasks the
baseline had passed. The net result was seven more successes.

An aggregate score compresses that result into a small improvement. It discards
the 125 changed outcomes.

The distinction matters whenever an optimization removes context, switches
models or changes a tool path. A population average can remain stable while the
identity of successful tasks changes underneath it. Users experience the
identity, not the average.

We found this pattern while reanalyzing the public artifact behind AgentDiet, a
trajectory-reduction method accepted at FSE 2026. The method has strong overall
results: across two models and two benchmarks, the paper reports 39.9 to 59.7
percent fewer input tokens and 21.1 to 35.9 percent lower computational cost
with comparable aggregate performance. Paired transitions expose which
workloads donated their success to produce that average.

## Four partitions, two directions

We downloaded the authors' 46.8 MB Figshare release, matched its published MD5,
and matched baseline and optimized trajectories by task. The held-out material
contained four partitions: two 200-task evaluation sets and two Multi-SWE sets
with 300 and 298 matched pairs. We excluded one unmatched item from each arm in
the final partition.

The transition table looked like this:

| Partition | Baseline to candidate pass rate | Pass to non-pass | Non-pass to pass | Matched pairs | Combined token change |
| --- | ---: | ---: | ---: | ---: | ---: |
| Evaluation, Claude | 64.5% to 66.5% | 7 | 11 | 200 | -31.4% |
| Evaluation, Gemini | 50.5% to 52.0% | 15 | 18 | 200 | -29.8% |
| Multi-SWE, default | 40.0% to 39.0% | 17 | 14 | 300 | -35.0% |
| Multi-SWE, Gemini | 21.7% to 22.7% | 20 | 23 | 298 | -52.1% |

Adding the discordant cells yields 59 regressions and 66 recoveries. Three
partitions improved slightly in aggregate. One lost a percentage point while
using 35 percent fewer combined tokens.

Trajectory reduction remains promising, but "quality held" is a population
statement. The treatment helped some tasks and hurt others, leaving deployment
policy to learn the boundary.

Seven net wins can contain 125 decisions that changed sign.

## Paired outcomes carry the information

Suppose two systems each solve 60 of 100 tasks. An unpaired report calls them
equal. Several realities fit that average:

- both systems solve the same 60 tasks;
- the candidate fixes ten failures and breaks ten successes;
- the two systems overlap on only 20 successes.

Those are different products. Stable behavior produces the first result. The
second creates a routing opportunity when task features predict the direction;
the last trades one capability profile for another.

A paired transition matrix preserves this structure:

| | Candidate passes | Candidate fails |
| --- | ---: | ---: |
| Baseline passes | stable success | regression |
| Baseline fails | recovery | stable failure |

The off-diagonal cells deserve priority. A regression names a task where the
optimization crossed the quality boundary. A recovery names evidence that the
same optimization can remove distraction or improve the agent's trajectory.

Statistical tests for paired binary outcomes, such as McNemar's test, operate on
these discordant cells. Because each task serves as its own control, the matched
design cancels some task difficulty that a raw comparison between arm totals
would retain.

## Token totals hide another mismatch

The public artifact records a primary coding agent and an analysis model used
by the optimization. Our `combined` column adds their reported token counts.
This is preferable to quoting only the primary agent, which would omit the
optimizer's overhead. It still does not make those tokens economically
equivalent.

Different models can have different prices, tokenizer behavior and cache
contracts. Ten thousand analysis-model tokens do not necessarily cost the same
as ten thousand primary-model tokens. The artifact also records no latency.
The combined reduction therefore measures token volume across the recorded
system, not catalog-priced spend or response time.

Cache behavior adds another missing term. Rewriting an old trajectory step may
reduce future context while invalidating a warm prefix; the outcome depends on
rewrite position, remaining calls and provider cache prices. A fixed artifact
cannot recover every provider-side cache event afterward.

These gaps do not make the token result useless. They set its claim boundary.
The artifact shows a large reduction in recorded token volume with modest
aggregate outcome movement. It cannot prove an invoice delta, a latency gain or
universal safety.

## Global promotion asks the wrong question

An average asks whether the candidate should replace the baseline everywhere.
The transition table points instead toward workloads where the candidate
preserves or improves outcome.

Start by stratifying the discordant pairs. Look for task family, repository
shape, model, trajectory length, tool mix and error mode. Only pre-decision
features can route. A label discovered after a successful trajectory may explain
the result but cannot guide future work.

This distinction prevents a common evaluation leak. Researchers inspect the
full task outcome, discover that long trajectories benefit, then build a rule
that depends on length accumulated after the optimization would already have
started. The policy looks predictive in retrospective data and is impossible to
apply prospectively.

Use the design split to choose candidate features and thresholds. Freeze them.
Evaluate once on a held-out split that played no role in those choices. When
several policies compete, account for selection or reserve a final untouched
set; every use of a benchmark to shape policy weakens it as a test.

## Regression budgets beat average floors

Many quality gates compare mean pass rate with a baseline floor. That catches a
large global collapse. It may approve severe subgroup damage if recoveries
elsewhere compensate.

A stronger gate has layers:

1. Aggregate non-inferiority. Overall performance stays within a
   pre-registered margin.
2. Paired regression limit. Pass-to-non-pass transitions stay below an
   explicit budget.
3. Critical-family floor. High-risk task families have their own minimums.
4. Cost per success. Resource reduction is divided by successful outcomes,
   not attempts.
5. Operational terms. Optimizer overhead, cache effects, latency and failure
   rates enter the result.

These gates answer different questions. A low mean delta can coexist with many
regressions, and limiting regressions says nothing about whether the cost gain
matters. Even a strong overall gain can violate a critical-family floor.

The regression budget also forces a product decision into the open. A 1 percent
mean improvement with 6 percent task churn may be acceptable for an optional
research assistant and unacceptable for a deployment agent. Statistics cannot
choose that tolerance. They can stop the average from hiding it.

## Report uncertainty at the task level

Agent runs are stochastic. One attempt per task gives broad workload coverage
and weak repeatability evidence. Repeating a tiny set estimates within-task
variance and generalizes poorly. Credible evaluation needs both dimensions.

Cluster uncertainty by task so repeated attempts do not masquerade as new
independent work. Publish counts alongside rates. "Seven regressions" conveys
more than "3.5 percent" when the denominator is 200. Show unmatched items,
crashes and exclusions in both arms.

Outcome grading deserves the same scrutiny. Repository tests and exact task
oracles carry clearer meaning than an unconstrained model judge, while partial
credit requires a declared pass threshold for paired transitions. Moving that
threshold after inspection changes which cells appear off the diagonal.

Finally, preserve negative partitions. The Multi-SWE default partition is the
most informative row in this analysis because its 35 percent combined-token
reduction coincided with a one-point pass-rate loss. A polished aggregate could
bury it inside the three partitions that improved. Keeping it visible reveals
the policy boundary research still needs to find.

## A minimum public result

For any context or routing optimization, publish enough data to reconstruct a
paired decision:

- matched task count and matching key;
- baseline and candidate successes;
- pass-to-fail and fail-to-pass transitions;
- per-task resource deltas, including optimizer overhead;
- model and harness versions;
- task-family breakdown chosen before evaluation;
- uncertainty method and repetition structure;
- unmatched, excluded and errored tasks;
- latency and cache telemetry, or an explicit statement that they are absent.

Then write the conclusion at the level the evidence supports. This artifact
makes trajectory reduction a strong candidate, but its safe domain varies by
workload. Selection and gating deserve continued work; universal activation
does not.

Averages are summaries, and this one was accurate. Only the transition matrix
showed what it had removed.

## Sources

1. Xiao et al., [Reducing Cost of LLM Agents with Trajectory Reduction](https://arxiv.org/abs/2509.23586), FSE 2026.
2. Xiao et al., [AgentDiet public artifact, version 2](https://doi.org/10.6084/m9.figshare.30073654.v2), Figshare, 2025.
3. Yao et al., [Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows](https://arxiv.org/abs/2605.27922), arXiv preprint, 2026.
4. Chen et al., [CoACT: Action-Preserving Observation Compression for Coding Agents](https://arxiv.org/abs/2607.02911), arXiv preprint, 2026.