---
title: "How to evaluate an AI agent optimization before rollout"
description: "An agent optimization needs two kinds of evidence: the result still meets the task's requirements, and the full cost or latency improves enough to justify the change. Neither a smaller prompt nor an a"
canonical: https://caveman.so/guides/agent-evaluations
last-updated: 2026-09-07
---

# How to evaluate an AI agent optimization before rollout

An agent optimization needs two kinds of evidence: the result still meets the task's requirements, and the full cost or latency improves enough to justify the change. Neither a smaller prompt nor an agent's own success message proves both.

Build the evaluation around work your team recognizes. A small, reproducible set of important tasks is a useful starting point.

## Write acceptance checks before tuning

Describe the final artifact and the conditions that make it unacceptable. For a code change, that includes required behavior, relevant tests, and scope. For extraction, it includes required values and schema. For a support answer, it includes factual correctness and whether the requested action occurred.

Prefer deterministic checks for facts a program can verify. Use human review or an LLM judge where judgment is needed, with a clear rubric and known limitations.

Do not let a broad score hide a hard failure. A patch that deletes a test to pass is rejected even if its explanation is excellent. A valid JSON object with the wrong customer ID is still wrong.

## Cover the cases an optimizer can break

| Candidate change | Cases worth including |
| --- | --- |
| Context compression | Required omitted detail, exact value, short input, recovery failure |
| Model routing | Tools, structured output, long context, fallback, difficult held-out tasks |
| Prompt shortening | Ambiguous instruction, exception, exact command or error |
| Gateway migration | Authentication, forbidden model, stream cancellation, retries |
| Framework adapter | Existing callbacks, tool side effects, interruption, missing usage |
| Conversation compaction | Earlier decision, unresolved work, resumed task |

Include ordinary successful work as well as failures. A suite made entirely from yesterday's incident can favor a fix that harms the rest of the workload.

## Preserve the starting state

Record the task input, repository or dataset revision, model, reasoning setting, agent version, tool configuration, and relevant environment. Use separate copies of the same initial workspace for baseline and candidate.

A continuation needs its starting history or checkpoint. A fresh session cannot stand in for a warm long-running session if cache reuse and memory are part of the claim.

For tools with side effects, use a staging environment and disposable fixtures. Reset only the test state you own between attempts. Do not replay production writes merely to create comparable traces.

## Keep tuning cases separate from acceptance cases

Use one set to design the prompt, routing policy, or compression behavior. Reserve other cases for evaluating the resulting candidate. Otherwise a system can learn the suite without improving the task family.

Version graders and expected outputs. If you change a rubric, rerun the baseline under that rubric. Scores produced by different judges or thresholds should not be plotted as one uninterrupted trend.

Keep a few hard failures visible as named cases. Their purpose is to prevent recurrence, not to disappear into an average.

## Run paired attempts

Alternate baseline and candidate order where practical. Repeat tasks enough to observe variation in your workload, and record sample count. Keep the same model and settings when the context path is the variable under test.

If the candidate changes the model too, call that out as a separate experiment. The result may still be useful, but it cannot isolate the effect of compression.

Record final artifacts, acceptance outcomes, complete usage, elapsed time, and errors. Save the source records that allow a reviewer to reconstruct the result.

## Count the work after the first call

Recovery, validation retries, repair turns, fallback models, and failed branches all contribute to task cost. A router that chooses a cheap model and then always escalates has bought two attempts.

Use provider charges where available. Otherwise retain usage and a dated price source, labeling the result as estimated. Missing prices or counters remain unknown. Local Caveman reports stay inferred.

```text
cost per accepted completion = spend on all attempts / accepted completions
```

When no attempt passes, the denominator is zero and the metric is undefined. Report the failure and spend instead of claiming a zero-cost result.

See [the measurement guide](/guides/measure-agent-cost) for cache normalization and subscription accounting.

## Inspect regressions before averaging

A candidate may help long logs and hurt short requests. That can justify a narrower policy with an explicit eligibility check. It does not justify dropping the losing cases from the report.

Open representative artifacts from both arms. Check whether the candidate avoided work the baseline completed, omitted a required caveat, or changed the tool path. Review slow and failed cases even when the aggregate improves.

For a proposed automated fix, inspect its scope and rerun cases it did not see during generation. A generated patch is a proposal until the relevant checks pass.

## Make the rollout rule explicit

Choose the required quality floor, acceptable latency, cost threshold, and rollback trigger from the task's needs. Some workflows require every hard check to pass; others can tolerate a measured error rate with human review.

Start with the eligible task family and keep the original path callable. Re-evaluate after meaningful changes to model, framework, agent version, or tool output.

Caveman's local tools can be evaluated using your existing tests and observability platform. A [private Platform pilot](/products/platform) should demonstrate the same chain from source evidence to case, intervention, and checked result.

## Keep a result someone else can review

Retain the baseline configuration, candidate change, cases, grader versions, raw outcomes, and rerun command. Explain missing evidence and limits on the conclusion.

For a concrete published example, inspect [CaveBench Wrap](https://github.com/JuliusBrussee/caveman/blob/main/docs/WRAP-BENCHMARK.md). Use its method and losing cases to inform your own experiment; its numbers are evidence about that benchmark, not a promised result for your agents.