Bring the whole target.
Compare a prompt, a model, an agent, an HTTP endpoint, or recorded outputs. Keep each target's input mapping explicit.
Evals & scenarios
Better agents.
Prove it.
Put prompts, models, agents, and workflows through the same cases. Inspect the failures. Make your next change earn its place.
Caveman Platform · private development
A good demo is one path through the problem.
Your users will find
the other thousand.
01 / Your cases. Your definition of good.
One grid.
Nowhere to hide.
Cases down the side. Variants across the top. Compare quality and cost, then open the exact result behind a pass or failure.
| Scenario | Baseline | Fast model | Guarded workflow |
|---|---|---|---|
| Valid tool callOutput contract | |||
| Ambiguous requestAsk before acting | |||
| Missing permissionAccess boundary | |||
| Noisy tool outputAnswer grounding | |||
| Long conversationContext retention | |||
| Provider timeoutFailure recovery | |||
| Passing cases | 5 / 6 | 2 / 6 | 6 / 6 |
Illustrative scored cases · select any cell · a cheaper failing variant does not earn promotion
02 / Test the behavior, not just the answer
Make scenarios repeatable.
Version datasets and expected behavior. Cover tool use, ambiguous input, access boundaries, and recovery paths with cases you can revisit.
Grade what matters.
Combine configured evaluators with human review. Inspect scores, reasons, and evaluator failures instead of trusting a single average.
03 / Close the loop with production
Today's failure.
Tomorrow's test.
Live quality monitors can feed failures into datasets. Compare candidates on those cases, then promote a passing evaluation into a gate for supported optimization paths.
Run receipt
Model calls2
Tool calls3
Stop reasonCompleted
Illustrative receiptMonitor the work.
Sample matching traffic with quality monitors. Keep coverage visible rather than implying every call was evaluated.
{
query: string,
limit: number
}Arguments validatedKeep the hard cases.
Carry trace references into an eval. Pin dataset and workbench versions so future results retain their context.
Quality
Cost
PolicyPending
Gate the change.
Use passing evaluation evidence to constrain promotion. A candidate with fewer tokens still has to meet the bar.
04 / Evidence before rollout
Pass the test.
Then test the rollout.
A passing eval is one part of the decision. Keep recorded replay, shadow observations, and live canary evidence distinct as a routing change moves forward.
Capture representative traffic before proposing a change.
Stage previewInteractive rollout illustration. Recorded outputs and replay are not proof of live-provider behavior. Gates and permitted rollout stages depend on the configured optimization.
05 / Inside the workbench
● ● ●Demo workspace
Real product UI · seeded demo workspace · catalog list-price subtotals
06 / Set the right bar
Are scenarios a separate product?
Scenarios are repeatable cases inside your evaluation datasets: inputs, expected behavior, tags, and references. Evals bring those cases together with targets and graders.
Can I compare an agent rather than a model?
The workbench supports agent, workflow, HTTP, model, prompt, and recorded targets. Available execution depends on the target configuration and the environment connected to your project.
Does a passing result mean production is safe?
It means the evaluated target passed the configured checks on those cases. Coverage, grader quality, human review, and controlled rollout still matter. Recorded replay and live-provider evidence remain separate.
Make every token count.
Ship the change.
Keep the standard.
Build a quality loop around your agents, then connect that evidence to the decisions that change them.