The average stayed flat. 125 tasks changed outcome.

An agent optimization was evaluated across 998 matched software-engineering tasks. It recovered 66 tasks the baseline had failed and broke 59 tasks the baseline had passed. The net result was seven more successes.

An aggregate score compresses that result into a small improvement. It discards the 125 changed outcomes.

The distinction matters whenever an optimization removes context, switches models or changes a tool path. A population average can remain stable while the identity of successful tasks changes underneath it. Users experience the identity, not the average.

We found this pattern while reanalyzing the public artifact behind AgentDiet, a trajectory-reduction method accepted at FSE 2026. The method has strong overall results: across two models and two benchmarks, the paper reports 39.9 to 59.7 percent fewer input tokens and 21.1 to 35.9 percent lower computational cost with comparable aggregate performance. Paired transitions expose which workloads donated their success to produce that average.

Four partitions, two directions

We downloaded the authors' 46.8 MB Figshare release, matched its published MD5, and matched baseline and optimized trajectories by task. The held-out material contained four partitions: two 200-task evaluation sets and two Multi-SWE sets with 300 and 298 matched pairs. We excluded one unmatched item from each arm in the final partition.

The transition table looked like this:

Adding the discordant cells yields 59 regressions and 66 recoveries. Three partitions improved slightly in aggregate. One lost a percentage point while using 35 percent fewer combined tokens.

Trajectory reduction remains promising, but "quality held" is a population statement. The treatment helped some tasks and hurt others, leaving deployment policy to learn the boundary.

Paired outcomes carry the information

Suppose two systems each solve 60 of 100 tasks. An unpaired report calls them equal. Several realities fit that average:

• both systems solve the same 60 tasks;
• the candidate fixes ten failures and breaks ten successes;
• the two systems overlap on only 20 successes.

Those are different products. Stable behavior produces the first result. The second creates a routing opportunity when task features predict the direction; the last trades one capability profile for another.

A paired transition matrix preserves this structure:

The off-diagonal cells deserve priority. A regression names a task where the optimization crossed the quality boundary. A recovery names evidence that the same optimization can remove distraction or improve the agent's trajectory.

Statistical tests for paired binary outcomes, such as McNemar's test, operate on these discordant cells. Because each task serves as its own control, the matched design cancels some task difficulty that a raw comparison between arm totals would retain.

Token totals hide another mismatch

The public artifact records a primary coding agent and an analysis model used by the optimization. Our combined column adds their reported token counts. This is preferable to quoting only the primary agent, which would omit the optimizer's overhead. It still does not make those tokens economically equivalent.

Different models can have different prices, tokenizer behavior and cache contracts. Ten thousand analysis-model tokens do not necessarily cost the same as ten thousand primary-model tokens. The artifact also records no latency. The combined reduction therefore measures token volume across the recorded system, not catalog-priced spend or response time.

Cache behavior adds another missing term. Rewriting an old trajectory step may reduce future context while invalidating a warm prefix; the outcome depends on rewrite position, remaining calls and provider cache prices. A fixed artifact cannot recover every provider-side cache event afterward.

These gaps do not make the token result useless. They set its claim boundary. The artifact shows a large reduction in recorded token volume with modest aggregate outcome movement. It cannot prove an invoice delta, a latency gain or universal safety.

Global promotion asks the wrong question

An average asks whether the candidate should replace the baseline everywhere. The transition table points instead toward workloads where the candidate preserves or improves outcome.

Start by stratifying the discordant pairs. Look for task family, repository shape, model, trajectory length, tool mix and error mode. Only pre-decision features can route. A label discovered after a successful trajectory may explain the result but cannot guide future work.

This distinction prevents a common evaluation leak. Researchers inspect the full task outcome, discover that long trajectories benefit, then build a rule that depends on length accumulated after the optimization would already have started. The policy looks predictive in retrospective data and is impossible to apply prospectively.

Use the design split to choose candidate features and thresholds. Freeze them. Evaluate once on a held-out split that played no role in those choices. When several policies compete, account for selection or reserve a final untouched set; every use of a benchmark to shape policy weakens it as a test.

Regression budgets beat average floors

Many quality gates compare mean pass rate with a baseline floor. That catches a large global collapse. It may approve severe subgroup damage if recoveries elsewhere compensate.

A stronger gate has layers:

1. Aggregate non-inferiority. Overall performance stays within a pre-registered margin.
2. Paired regression limit. Pass-to-non-pass transitions stay below an explicit budget.
3. Critical-family floor. High-risk task families have their own minimums.
4. Cost per success. Resource reduction is divided by successful outcomes, not attempts.
5. Operational terms. Optimizer overhead, cache effects, latency and failure rates enter the result.

These gates answer different questions. A low mean delta can coexist with many regressions, and limiting regressions says nothing about whether the cost gain matters. Even a strong overall gain can violate a critical-family floor.

The regression budget also forces a product decision into the open. A 1 percent mean improvement with 6 percent task churn may be acceptable for an optional research assistant and unacceptable for a deployment agent. Statistics cannot choose that tolerance. They can stop the average from hiding it.

Report uncertainty at the task level

Agent runs are stochastic. One attempt per task gives broad workload coverage and weak repeatability evidence. Repeating a tiny set estimates within-task variance and generalizes poorly. Credible evaluation needs both dimensions.

Cluster uncertainty by task so repeated attempts do not masquerade as new independent work. Publish counts alongside rates. "Seven regressions" conveys more than "3.5 percent" when the denominator is 200. Show unmatched items, crashes and exclusions in both arms.

Outcome grading deserves the same scrutiny. Repository tests and exact task oracles carry clearer meaning than an unconstrained model judge, while partial credit requires a declared pass threshold for paired transitions. Moving that threshold after inspection changes which cells appear off the diagonal.

Finally, preserve negative partitions. The Multi-SWE default partition is the most informative row in this analysis because its 35 percent combined-token reduction coincided with a one-point pass-rate loss. A polished aggregate could bury it inside the three partitions that improved. Keeping it visible reveals the policy boundary research still needs to find.

A minimum public result

For any context or routing optimization, publish enough data to reconstruct a paired decision:

• matched task count and matching key;
• baseline and candidate successes;
• pass-to-fail and fail-to-pass transitions;
• per-task resource deltas, including optimizer overhead;
• model and harness versions;
• task-family breakdown chosen before evaluation;
• uncertainty method and repetition structure;
• unmatched, excluded and errored tasks;
• latency and cache telemetry, or an explicit statement that they are absent.

Then write the conclusion at the level the evidence supports. This artifact makes trajectory reduction a strong candidate, but its safe domain varies by workload. Selection and gating deserve continued work; universal activation does not.

Averages are summaries, and this one was accurate. Only the transition matrix showed what it had removed.

Sources

1. Xiao et al., Reducing Cost of LLM Agents with Trajectory Reduction, FSE 2026.
2. Xiao et al., AgentDiet public artifact, version 2, Figshare, 2025.
3. Yao et al., Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows, arXiv preprint, 2026.
4. Chen et al., CoACT: Action-Preserving Observation Compression for Coding Agents, arXiv preprint, 2026.