Token efficiency you can recompute.

CaveBench measures how much an optimizer cuts cost while holding quality, then signs the run. Re-run the open harness on the open data and the signature recomputes, byte for byte.

80.4%

fewer tokens

9,609 → 1,882

100%

quality kept

floor ≥ 99%

100%

byte-recoverable

lossless round-trip

74.8%

cost saved

at held quality

signedf09748bc5b…ff7cc1key cavebench-localcatalog 2026-07-10basis inferred

Where the savings come from.

Reported per content type, because that is where the story lives: Caveman compresses structured payloads losslessly and recovers them byte-for-byte, and declines to compress free prose rather than touch the bytes.

content typetoken reductioncost savedqualitybyte-recov.

Terminal sessions

byte-safe

91%

86%100%100%

Repetitive output

byte-safe

91%

86%100%100%

Build & run logs

byte-safe

91%

89%100%100%

Tool & JSON output

byte-safe

86%

84%100%100%

Tool schemas

byte-safe

71%

58%100%100%

HTML

byte-safe

69%

59%100%100%

Source code

byte-safe

57%

52%100%100%

Tabular JSON (TOON)

lossless to model

21%

14%100%100%

Free prose & chat

passthrough

none

—100%100%

Byte-recoverability is 100% on every compressed class — a column no lossy compressor can score on, since it cannot recover the tokens it dropped.

Cost per correct task

At identical 100% task success, Caveman costs 3.4× less per correct task — the savings come from the input tokens it never sends.

raw passthrough

389µ$ / 499 tok

caveman

114µ$ / 132 tok

How it's measured.

The construct is inferred token reduction at held quality on representative tasks — not raw byte savings, provider billing, or verified live-traffic savings.

Sweep the curve, never one number

Compression versus quality is non-monotonic, so the harness sweeps a grid of targets and reports the curve, stratified by context length — never a single context-free percentage.

Two ratios, for two reasons

A word-ratio that compares across providers, and an engine-token ratio with its tokenizer disclosed. Provider-reported usage is separate; neither benchmark ratio is invoice truth.

Decontaminate, then test significance

A 13-gram overlap filter removes leaked items, and Versus deltas are reported as a mean with a 95% interval — a delta that straddles zero is not a win.

This run grades quality offline against deterministic answer keys (exact_match, json_schema, tool_sequence) and estimates cost from the dated catalog rather than a billed invoice — 15/15 tasks passed, both ways. Savings stay inferred and non-billable; a benchmark or active rollout never enters the verified ledger. Single-number compression claims — say, 20× at near-zero loss — do not survive this harness, which is why it sweeps the curve and reports intervals.

The receipt you can reproduce.

Every run is wrapped in an Ed25519-signed, hash-chained receipt carrying the token counts, the cost, the eval-gate result, and the exact formula and catalog versions. Public leaderboards publish a number; CaveBench publishes the number and the signature that recomputes it.

caveman.receipt.v3signed

day: 2026-06-21
seq: #1
requests: 15
tokens before: 10,044
tokens after: 2,319
benchmark cost: 1957µ$
eval gate: 15 pass · 0 fail
verified savings: $0; honest zero
formula: cavebench.self.v1

report hash

receipt hash

prev receipt∅ genesis

Ed25519 signaturekey cavebench-local

Reproduce it

One deterministic command regenerates the golden and its signature. Same inputs, same toolchain, same catalog snapshot → an identical receipt hash.

$ make bench-golden

verifies this run

trust anchor · production

Trust model

What this claims is reproducibility, not neutral attestation: the signing key is founder-held and repo-pinned. Receipts carry their key_id, so historic ones stay verifiable through rotation. Custody & neutrality policy: docs/cavebench/GOVERNANCE.md.

What's live, and what's next.

CaveBench-Self is live and signed — every number above is from it. Two stages remain, blocked on the work below, not on the result.

CaveBench-Selflive · signed
Caveman grades and signs its own savings on a fixed fixture suite — every number on this page.
CaveBench-Versusnot yet published
The same harness head-to-head against rival compressors. Opens once the rival adapters and the canonical open suite are configured.
CaveBench-Publicnot yet published
A standing leaderboard ranking any agent or model by cost-per-correct-task. Opens after the first signed full run on a decontaminated open suite.

Caveman runs the benchmark it sells, and judges its own entry under the same rules — same suite, same contamination and significance checks, no privileged conditions. Key custody and the neutrality stance: docs/cavebench/GOVERNANCE.md.

Leaderboard status

The harness is open. The receipt is reproducible.

Clone the repo, run make bench-golden on the open data, and compare your signed receipt against the one above.

Browse the harness Leaderboard status