Labs / research
CaveBench methodology
Abstract
Every compression vendor quotes a percentage. Almost none report what answer quality was while the percentage was being earned, and none that we know of let a skeptic recompute the number from open materials. CaveBench is a benchmark for token-efficiency systems built on three commitments: savings figures are conditioned on a pre-registered quality floor, results are statistically qualified across seeds, and every published run carries a signed, recomputable receipt. This report defines the metrics, states the design rules and the evidence behind them, describes the attestation model together with its limits, and gives the benchmark's current status honestly: the self-benchmark phase runs today as a merge gate on our own system, while the head-to-head and public-leaderboard phases are built but deliberately not live.
1. Motivation
A savings number with no quality floor is unfalsifiable. Anyone can cut 90 percent of tokens if the answers are allowed to get worse, so a bare percentage says nothing about whether a system did useful work. The claim becomes testable only once it is conditioned: this much cost removed while quality stayed at or above a stated floor. That version of the claim can lose, which is what makes it worth winning.
CaveBench is the benchmark we would want to exist if a competitor were running it against us. The design goal throughout is that a third party can re-run any published result and arrive at the same bytes.
2. Metrics
The headline metric is savings at held quality: the cost reduction achieved while quality retention stays at or above a floor Q. The default floor is Q = 0.99 of the baseline eval pass rate, pre-registered in a committed configuration file (bench/floors.json). It is the same floor the product's eval gate enforces during rollout, so the benchmark claim and the shipped behavior are one number rather than two.
Three quantities appear on every report.
| Metric | Definition | What it rules out |
|---|---|---|
| Savings at held quality | Cost reduction with retention at or above 0.99 of the baseline pass rate | Quality silently traded for the headline |
| Cost per correct task | Total cost divided by correct completions | Cheap wrong answers scoring well |
| Tokens per resolved task | Total tokens divided by resolved tasks | Verbosity hiding inside a high pass rate |
The two ranking scalars divide by successes rather than attempts, because a cheap wrong answer is not efficiency.
3. Design rules
Sweep, never point-sample. The relationship between compression ratio and task quality is not monotonic; moderate compression can improve long-context accuracy by removing distractors, consistent with the degradation observed when relevant material sits buried in long irrelevant context (Liu et al., 2024). A system evaluated at one hand-picked operating point can therefore look arbitrarily good. The harness sweeps a grid of compression targets and publishes the full savings-versus-retention curve.
Segment, never blend. Results are reported per content type and per safety class, with byte-safe structural compression separated from lossy token-dropping methods in the LLMLingua family (Jiang et al., 2023), because the two make different promises. On structured payloads, recoverable structural compression wins honestly; on free prose, lossy methods often compress further, and the report shows that column rather than hiding it.
Qualify statistically. Every gateway run reports retention and savings as a mean with a 95 percent confidence interval over multiple seeds. A versus-delta whose interval straddles zero is published as not significant. A single point dressed up as a range is not accepted.
4. Attestation
Every CaveBench run emits an Ed25519-signed, hash-chained receipt recording the formula version and the dated pricing-catalog snapshot used to compute costs. The signing public key is committed in the repository as a pinned trust anchor. A skeptic re-runs the open harness on the open data and obtains either a byte-identical signed result or a provable discrepancy.
We are precise about what this model claims. It claims reproducibility, not third-party neutrality: the private key is founder-held, rotating it is a reviewed change to the committed anchor rather than a silent swap, and escrow with a neutral party is a planned step we do not pretend to have taken.
5. Gaming resistance
The governance is standard and written down. Each model version gets one locked submission. Evaluation budgets are symmetric across entrants, so nobody buys extra attempts. The public phase scores against a hidden, rotating held-out split published only as a hash. A 13-gram overlap filter between task items and known training corpora, following the decontamination protocol of Brown et al. (2020), blocks publication on any overlap; task freshness is paired with explicit decontamination rather than trusted on its own.
Caveman's own product runs under identical rules and identical contamination checks as any other entrant.
6. Status
Phase 1, CaveBench-Self, is built and gated: it runs in this repository with a byte-reproducible golden report and a per-pull-request floor gate, so a regression in our own savings blocks the merge that caused it. Phase 2, head-to-head against other systems, and Phase 3, a public leaderboard, are built to launch-ready and report themselves blocked until a canonical open task suite and configured rival adapters exist. There is no live public leaderboard today, and nothing in this report should be read as implying one.
One rule carries over from the rest of the stack: a benchmark result is inferred evidence. Nothing on any leaderboard becomes verified savings until an optimizer runs in active mode on real traffic and the ledger records the dollars (see The zero-dollar dashboard).
References
- Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33.
- Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023.
- Liu, N. F., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12.