---
title: "Agent observability: connect requests, cost, and completed work"
description: "A useful agent trace explains a piece of work. It shows which agent acted, what the model and tools did, where the run failed or repeated itself, and what the task produced. Token totals become useful"
canonical: https://caveman.so/guides/agent-observability
last-updated: 2026-09-07
---

# Agent observability: connect requests, cost, and completed work

A useful agent trace explains a piece of work. It shows which agent acted, what the model and tools did, where the run failed or repeated itself, and what the task produced. Token totals become useful when you can follow them back to those records.

Start with one workflow whose result you understand. Collect enough evidence to investigate it before building a company-wide cost dashboard.

## Define the task boundary

A provider request is usually smaller than the job the user asked for. A support answer may involve retrieval, several model calls, a failed tool, and a final check. A coding task may continue across sessions before a patch is accepted.

Give the task a stable identifier and give each attempt its own identifier. Preserve relationships across agents, retries, and resumed work. Do not group unrelated requests solely because they occurred near each other in time.

The final artifact matters too. Link the answer, patch, or structured result to the task where your application can prove that relationship. A repository name alone does not establish which change a session produced.

## Collect the minimum useful record

| Field family | Purpose |
| --- | --- |
| Task, attempt, trace, and parent IDs | Reconstruct related work without merging separate attempts |
| Agent, workflow, environment, and owner | Attribute activity to an operational unit |
| Provider, model, and deployment | Identify the inference path and pricing context |
| Start, finish, status, and error class | Explain latency and failure |
| Input, output, cache, and reasoning usage | Preserve the provider's accounting evidence |
| Final artifact and acceptance result | Distinguish work attempted from work completed |
| Source and completeness | Show which conclusions the record can support |

Collect content only when the investigation needs it and the application's data policy permits it. Timing and usage metadata can answer many questions without copying prompts or source code into another store.

A trace of the original tool output may differ from the model-facing compressed view. Label the stage if both are recorded.

## Normalize usage without inventing evidence

Providers do not all mean the same thing by input tokens. Cache reads may be a subset of input in one API and a separate field in another. Reasoning tokens may be included in output. Read the source contract before summing.

Unknown is a valid state. Missing cache fields should not become measured zeros. A model absent from your price catalog should remain unpriced. A request with incomplete usage should not receive a precise-looking cost.

This example describes a reporting shape, not a Caveman ingestion API:

```json
{
  "task_id": "support-case-42",
  "attempt_id": "attempt-2",
  "usage_source": "provider_response",
  "accounting_status": "incomplete",
  "cost_usd": null,
  "acceptance": "passed"
}
```

The task can pass while accounting remains incomplete. Those facts should coexist instead of one overwriting the other.

## Reconcile before making a dashboard claim

Choose a small time window and compare application records, gateway logs, and provider usage. Explain discrepancies from retries, sampling, delayed ingestion, duplicate instrumentation, or cache semantics.

Two wrappers around the same provider client can produce two spans for one call. A retry performed below your callback can produce a billed call with no matching application event. Neither issue is fixed by choosing a nicer chart.

Measure coverage explicitly. For example, report the number of calls with complete usage and the number with a known owner. Do not call a partial ledger the total bill.

## Investigate one expensive task

Start from the accepted result and trace backward. Find the largest repeated input, slow branch, or failed attempt. Ask what change could plausibly improve that path.

A long tool result suggests [compression](/guides/prompt-compression) or a more focused tool query. Repeated validation failures suggest a schema or tool-contract fix. Easy tasks on an expensive model suggest [a model evaluation](/guides/model-routing). Duplicated research suggests an orchestration change.

Write a hypothesis that a test can reject. “This search result is repeated on six turns; a smaller view should preserve the required answer with fewer total input tokens” is specific enough to evaluate.

## Turn the trace into an evaluation case

Preserve the permitted input, environment, expected result, and grader. Keep the source trace reference with the case. Use disposable fixtures for tools with external side effects.

Run baseline and candidate with the same acceptance rule. Include failed attempts and complete usage. Rerun the case to establish whether the proposed change works.

The [evaluation guide](/guides/agent-evaluations) covers case design and release decisions. The [measurement guide](/guides/measure-agent-cost) covers cost per accepted completion.

## Pick collection and review tools by the workflow

[Langfuse](/compare/langfuse), [LangSmith](/compare/langsmith), [Braintrust](/compare/braintrust), [Phoenix](/compare/arize-phoenix), [AgentOps](/compare/agentops), and [PromptLayer](/compare/promptlayer) have different strengths and runtime dependencies. Compare the ingestion, evaluation, prompt, and deployment contracts you need.

Caveman's public local tools can reduce eligible context while those systems remain in place. Caveman Platform is in private development and focuses on connecting agent activity, cost, work, and tested improvements. For a pilot, agree on supported ingestion and prove one complete evidence path before widening collection.

## Keep the next action close to the record

A good review ends with a bounded experiment and an owner. Retain the trace, test case, proposed change, and rerun result together. Mark unsupported conclusions clearly, especially when content or provider usage is missing.

That gives the team a repeatable route from an expensive task to a change it can inspect and accept.