---
title: "Prompt caching and compression: reduce agent cost without losing reuse"
description: "Prompt caching reuses provider work for repeated context. Compression changes the context itself. Both can reduce cost, but applying them together requires a complete-task comparison: a shorter reques"
canonical: https://caveman.so/guides/prompt-caching
last-updated: 2026-09-07
---

# Prompt caching and compression: reduce agent cost without losing reuse

Prompt caching reuses provider work for repeated context. Compression changes the context itself. Both can reduce cost, but applying them together requires a complete-task comparison: a shorter request can become more expensive if it loses a valuable cache hit.

Start by finding out whether your current requests get cache reuse. Do that before rewriting prompts, moving models, or adding a second context policy.

## Distinguish three different caches

| Cache | What is reused | What to check |
| --- | --- | --- |
| Provider prompt cache | Processing of an eligible repeated prefix or cached context | Provider usage, model rules, reads, writes, and retention |
| Gateway response cache | A prior answer | Correctness, freshness, authorization scope, and invalidation |
| Application retrieval cache | Tool or search results | Source freshness and whether the model still processes the result |

A gateway cache hit can avoid a model call entirely. A provider cache hit still involves a model call and output generation. Reusing a database result in your application does not automatically mean the model's input was cached.

Name the layer in your experiment. Otherwise two dashboards can both show a “cache hit” while describing different work.

## Keep reusable content stable

Put stable instructions and shared reference material before frequently changing task content where the API supports that structure. Avoid inserting timestamps or random identifiers at the beginning of every request unless the model needs them there.

Tool definitions and earlier conversation content can also affect reuse. Reordering tools or rewriting history may change the shared prefix even when the visible user question barely changes.

Provider rules differ by API and model. [OpenAI's caching guide](https://developers.openai.com/api/docs/guides/prompt-caching) explains prefix matching, breakpoint behavior, and model-specific limits. [Claude's guide](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) covers cache controls and separate read and creation usage. [Gemini's guide](https://ai.google.dev/gemini-api/docs/caching) distinguishes API support for implicit and explicit caching. Check the exact model and endpoint you run rather than copying a threshold from another provider.

## Record writes as well as reads

A warm read can be cheap while creating or storing the cache has its own cost. Include the first request and the reuse pattern you expect in production.

For Claude's documented usage contract, total input is the sum of fresh input, cache-read input, and cache-creation input. Other APIs can report cached tokens as a subset of total input. Normalize according to the source contract and avoid adding a subset twice.

Keep missing usage unknown. A framework that reports no cache field has not established that no cache was used.

## Use a small experiment matrix

Run four configurations against the same checked tasks:

| Arm | Context | Cache state |
| --- | --- | --- |
| A | Original | First use |
| B | Original | Repeated stable prefix |
| C | Candidate compression | First use |
| D | Candidate compression | Repeated candidate prefix |

Keep model, reasoning, tool definitions, and task input fixed within the comparison. Separate provider caching from any gateway response cache. Confirm that each arm actually reached the intended path.

Repeat the sequence with the reuse interval your application expects. A rapid local loop and a task resumed much later can have different cache behavior.

## Understand the shorter-but-costlier case

Consider an illustrative price model where uncached input costs ten times as much as cached input. These are invented units for explanation, not current provider prices:

```text
Original: 10,000 cached tokens × 1 unit = 10,000 units
Candidate: 2,000 uncached tokens × 10 units = 20,000 units
```

The candidate uses 80% fewer input tokens in that call and costs twice as much for input. Over several turns it may build a useful new cache, or it may keep changing the prefix and never recover the loss. That is why the task boundary matters.

Include output, cache writes, retries, and recovery in the actual calculation. Use current provider prices or charges, with their source recorded.

## Add Caveman with an explicit mode

Caveman's supported wrappers default to compression. A standalone listener without explicit configuration starts in record mode. Check your active mode with the documented setup and status commands before interpreting results.

The local proxy also has provider-cache planning behavior on supported paths. Planner metadata and model-visible context changes are different operations. Read [proxy configuration](https://docs.caveman.so/docs/proxy/configure) for the exact controls and keep only one planned change in the first trial.

Do not compress an already stable prefix merely because a local ratio looks attractive. Target expensive eligible context and compare its effect on later reuse.

## Decide from your reuse pattern

For one-off tasks, cold cost may dominate. For long agent sessions, preserving a growing stable prefix can matter more. For repeated application queries, response-cache correctness may be the first issue to solve.

Use [the measurement guide](/guides/measure-agent-cost) to price complete attempts and [prompt compression](/guides/prompt-compression) to verify recovery. Keep the configuration that improves accepted work under your real cache pattern.