---
title: "Test Caveman with your Braintrust evaluation baseline"
description: "Use your existing Braintrust experiment as the baseline for a Caveman trial. Keep its dataset and scorer versions fixed while you change the candidate agent's context path. This establishes whether Ca"
canonical: https://caveman.so/switch/braintrust
last-updated: 2026-09-07
---

# Test Caveman with your Braintrust evaluation baseline

Use your existing Braintrust experiment as the baseline for a Caveman trial. Keep its dataset and scorer versions fixed while you change the candidate agent's context path. This establishes whether Caveman helps before you consider moving evaluation infrastructure.

A broader migration to Caveman Platform requires private access and a supported ingestion plan. No public command imports an entire Braintrust project into Caveman.

## Preserve the experiment's inputs and meaning

Record dataset versions, expected outputs, scorer implementation, thresholds, prompt versions, model settings, and tool configuration. Retain source experiment and production-log links for reviewers.

Check whether scorers depend on hosted functions, external services, or an LLM judge. Save those dependencies and include their cost in the experiment budget. A changed judge can move a score even when the agent is unchanged.

Use [Braintrust's current documentation](https://www.braintrust.dev/docs) for supported exports and SDK behavior. Do not treat a downloaded score table as a complete reproduction of the experiment.

## Select cases that can expose a regression

Include normal work, long tool output, a failed tool, a repeated lookup, and a question requiring an earlier detail. For coding tasks, start from the same commit or disposable workspace and check the produced patch.

Keep a held-out set that was not used to tune compression or prompts. Separate deterministic checks from subjective scoring. A compilation failure should not disappear inside an average judge score.

Write down the result you would accept: for example, every required check passes and median full-task input decreases without an unacceptable latency increase. Choose the threshold from your workload instead of adopting a vendor benchmark as a promise.

## Add Caveman as one candidate

For a supported coding-agent launcher:

```bash
npm install -g @caveman-ai/cli
caveman setup --install
caveman doctor claude
caveman claude
```

Run direct and Caveman sessions independently. Keep the same task and model. Record all recovery and retry calls, then attach the final artifact and validation result to the experiment through your existing workflow.

For a standalone tool-result check, use a saved fixture:

```bash
caveman shrink --file ./fixtures/tool-output.json
```

Inspect the output and any recovery handle before integrating it. A fixture test shows what happened to that content; it does not establish agent quality or billed savings.

## Compare evidence before totals

| Result | What to reconcile |
| --- | --- |
| Task success | Same grader version and acceptance rule |
| Usage | Same run boundary, retries, recovery, and cache fields |
| Latency | Full task duration and time spent grading |
| Cost | Provider usage versus catalog estimate versus invoice |
| Failure | Raw status and artifact remain available |

Use [the cost guide](/guides/measure-agent-cost) to report unsuccessful attempts and missing prices. Do not silently drop them from a candidate's denominator.

## Move a Platform workflow only after mapping it

For private Platform access, agree on the event source and fields Caveman supports. Start with one staging workflow and a small evaluation set. Confirm parent relationships, task identity, usage, failures, content policy, and grader provenance.

Keep Braintrust prompt delivery and production evaluation jobs in place until their replacements pass their own checks. An exporter change does not move prompt versions, schedules, or scorer code.

Have a reviewer follow one candidate result back to the original task and inspect the proposed improvement. That is a more useful acceptance check than matching dashboard colors or aggregate totals.

## Roll back

Restore the direct agent launch or original collection configuration. Keep the frozen Braintrust experiment callable throughout the trial. Retain historical records and credentials until dependent jobs and reviewers no longer need them.

The [comparison](/compare/braintrust) explains when Caveman is an addition to Braintrust and when a broader Platform evaluation is worth the work.