---
title: "How to measure AI agent cost per completed task"
description: "An optimization earns its place when the same work gets done with less total cost. Counting the bytes removed from a prompt cannot establish that. The model might ask for the original, make another at"
canonical: https://caveman.so/guides/measure-agent-cost
last-updated: 2026-09-07
---

# How to measure AI agent cost per completed task

An optimization earns its place when the same work gets done with less total cost. Counting the bytes removed from a prompt cannot establish that. The model might ask for the original, make another attempt, or lose a cache hit.

This guide gives you a small experiment you can run before changing your default agent setup. Use it for compression, model routing, shorter instructions, or a gateway migration.

## Define finished before you run

Choose a task with an outcome you can check without trusting the agent's own report. For a code change, use the patch requirements and the relevant tests. For log analysis, define the exact incident ID and error that the answer must contain. For a structured extraction task, compare the output against a saved expected result.

Write down the rejection conditions too. A patch that passes tests by deleting the failing assertion is a failure. A shorter summary that omits the only important warning is a failure. Keep the grader the same for both configurations.

## Freeze one baseline and one change

Record the repository revision, starting files, prompt, model ID, reasoning setting, agent version, and relevant tool configuration. Save the initial conversation state if you are measuring a continuation. An empty session and a session with a warm prefix answer different questions.

Change one thing for the candidate. For a local Caveman trial, that can be the normal agent launcher versus its Caveman wrapper:

```bash
# Baseline, fresh session
claude

# Candidate, another fresh session
caveman claude
```

Use the same task instructions in both. Keep any output-shortening skill either enabled in both arms or explicitly part of the candidate. Otherwise the experiment silently compares two changes.

If the task writes files, run it in separate copies of the same starting checkout. Do not let the second arm inherit the first arm's patch, generated files, or test cache changes that affect the task.

## Record a complete run

A useful result row contains:

| Field | Why it matters |
| --- | --- |
| Task ID and arm | Pairs the two runs |
| Model, provider and settings | Identifies what was purchased |
| Fresh input, cached input, cache writes, output | Separates usage with different prices |
| Retries and recovery calls | Keeps the overhead inside the result |
| Total elapsed time | Catches a cheaper but unusably slow path |
| Correctness result | Prevents cheap failures from looking like wins |
| Cost source | Distinguishes a provider charge from an estimate |

Some APIs report cached tokens as a subset of total input; others expose separate counters. Normalize according to that provider's contract before doing arithmetic. Do not add a subset twice. Keep missing usage unknown.

## Price the whole attempt

If you have provider charges for every call, sum them across the attempt. If you only have usage counts and a price catalog, label the result as an estimate and record the catalog date. Include paid judge calls when they are part of the production workflow you are comparing.

```text
attempt cost = sum of all model-call charges in that attempt
cost per completed task = total spend across all attempts / accepted completions
```

If no attempt completes, cost per completed task is undefined. It is not zero. Keep the failed attempts in the total when comparing a group of runs. Deleting expensive failures rewards unreliable configurations.

A subscription bill needs a separate reading. Fewer tokens in one session do not automatically lower a fixed monthly fee or increase a quota. Report the measured usage change without converting it into invented cash savings.

## Repeat and inspect the losing cases

Alternate baseline and candidate run order so provider conditions do not always favor the same arm. Repeat enough tasks to cover your actual payloads: tiny outputs, long logs, structured data, and code changes that need detail. Do not select only inputs that a compressor recognizes.

Look at each task before averaging. A candidate that helps large logs but loses on short requests may deserve a narrower rollout. Report sample count and variation. A small trial can identify a useful next step; it cannot prove a universal percentage.

## Decide what to change

Adopt the candidate for the workflows where correctness holds and the total result improves. Keep the original path available. Repeat the comparison when the model, harness, or tool-output format changes enough to invalidate the result.

Caveman's local reports stay `inferred`. A controlled paired benchmark is evidence about that experiment. Provider-confirmed savings on live traffic require a stronger causal comparison. Preserve those distinctions in dashboards and in anything you send to a customer.

Read [how Caveman counts](https://docs.caveman.so/docs/counting), [local proxy reporting](https://docs.caveman.so/docs/proxy/tokens), and the [CaveBench methodology](/labs/articles/cavebench-methodology) for the product's accounting rules.