---
title: "The 47 tokens your agent should never summarize"
description: "Imagine an agent carrying one standing instruction: never email a document to"
canonical: https://caveman.so/news/the-47-tokens-your-agent-should-never-summarize
last-updated: 2026-08-13
---

# The 47 tokens your agent should never summarize

Imagine an agent carrying one standing instruction: never email a document to
an address outside the company domain. It follows the rule for an hour. The
session grows, old turns are compacted into a summary, and the work continues.
Asked again, the same model sends the same document to an external address.

Nothing changed in the model or request. Its memory manager had deleted the
rule.

A 2026 preprint calls this failure "governance decay." Across 1,323 benchmark
episodes spanning seven model families, agents violated none of the tested
policies while those policies remained in full context. After one compaction,
the pooled violation rate reached 30 percent and peaked at 59 percent for two
model families. A small protected policy buffer restored the benchmark rate to
zero. Its average size was about 47 tokens.

Forty-seven is a result from one benchmark, not a universal budget. The durable
lesson is architectural: some state has no business entering a lossy summary.

## Compaction edits the agent's constitution

Long-running agents eventually exceed a practical context budget. Harnesses
respond by truncating old turns or asking a model to rewrite them. Selective
eviction offers another route. The compacted state becomes the history used for
every later decision.

That rewrite usually optimizes for task continuity. It keeps recent progress
and the next likely action. A standing rule
loaded forty turns ago looks less relevant, although task relevance and
authority are different properties. "Send the report" describes current work;
"Never send outside this domain" constrains every future version of that work.

The distinction is easy to miss because a good summary reads coherently. There
is no parse error when a prohibition disappears. Later actions remain fluent
and locally sensible. The agent itself cannot provide an independent check,
since its judgment is now conditioned on the damaged summary.

The governance-decay study isolates this mechanism with deterministic tool-call
grading. Each scenario establishes a policy, fills the context with ordinary
work, compacts it, then issues a request whose natural completion violates the
policy. The grader inspects the final action, such as an external recipient or
a production database command. No model judge decides whether the action was
safe.

When a policy survived the compacted summary, the observed violation rate was
zero. After deletion it reached 38 percent, and a second scoring method pointed
the same way. Generic long-context confusion does not explain that split.

## Soft rules disappear first

Model alignment already gives certain restrictions a home in the weights.
Requests to expose an SSH key or disable security controls may trigger a refusal
even after a local policy vanishes. Company-specific rules lack that backup.

The study separated four familiar safety norms from five organization-specific
policies involving region, language, spending, reporting channel and external
email. Compaction raised violations by 50 percentage points on the soft local
rules and six points on the hard norms. The decay gap was 8.3 times.

This explains why ordinary safety testing can miss the problem. A harness may
retain refusals for obviously dangerous actions while losing the rules that
make an otherwise normal action prohibited inside one deployment. The model
still looks safe on generic tests. It has forgotten what this organization
means by safe.

Constraint placement also mattered. Preserved system instructions survived in
the study. Standing user instructions, retrieved memory and tool-returned policy
text remained exposed to compaction. Real systems use all three channels for
dynamic governance because permissions, customer terms and incident controls
change faster than model weights or global prompts.

An agent can remember the assignment and forget the authority under which it acts.

## Summaries optimize the wrong loss function

Another 2026 preprint studied a less dangerous version of the same information
bottleneck. A research agent examined a closed inventory of flights and hotels,
then handed compressed state to a booking agent. The second agent had no access
to the original inventory. Narrative summaries produced the smallest payload
and only 0.48 feasibility accuracy. Schema-constrained JSON reached 0.96.

Fluency is the wrong compression objective for numeric limits and categorical
constraints. A sentence can sound faithful while rounding a price, dropping an
exception or merging conditions; a typed field exposes whether the compactor
retained the value.

Structure helps with task state. It does not make lossy transformation suitable
for authority. A JSON field can still contain a paraphrased rule whose meaning
shifted. The safer design separates state by required fidelity:

| State class | Treatment | Example |
| --- | --- | --- |
| Authority | Preserve verbatim and integrity-check | permissions, prohibitions, approval limits |
| Exact evidence | Store outside context and cite by stable ID | command output, source excerpt, signed fact |
| Working state | Encode into validated structure | objective, decisions, files changed, blockers |
| Narrative residue | Summarize or evict | superseded discussion, redundant explanation |

This is a storage policy, not a prompt trick. Each class gets a different
failure contract.

## What pinning does

Constraint pinning keeps governed rules out of material handed to the
compactor. After each rewrite, the harness injects the protected buffer into
the new context and verifies its presence. In the ConstraintRot benchmark,
that buffer averaged about 47 tokens, less than 0.5 percent of the tested
production-scale compaction context. Violation returned to zero in both the
ordinary compaction and tested attack conditions.

A minimal implementation has four parts:

1. Mark authoritative constraints when they enter the session. Record their
   origin and exact bytes.
2. Exclude them from truncation, summarization and model-selected eviction.
3. Reinsert them after every compaction in a stable, high-authority location.
4. Compare the resulting bytes or digest with the protected source. Reject the
   rewrite if any required item is missing.

The fourth part turns a writing task into an invariant. Beautiful prose can
still fail it, in which case the original history remains active.

Pinning also narrows an injection surface. Tool outputs and retrieved documents
can contain instructions aimed at the summarizer itself. The paper's
compaction-eviction attack asks the summarizer to omit a legitimate policy.
Optimized variants defeated every evaluated model. A compactor cannot delete
state it never receives authority to rewrite.

## Pinning does not enforce the rule

Keeping a policy visible improves model behavior. It does not create a security
boundary around a dangerous tool.

Recipient policy still belongs in the email connector, and deployment
permissions need enforcement outside the language model. Deterministic
authorization remains the final gate.

Pinning solves a different problem: policy carriage. It ensures the model sees
the current constraint while planning and explaining its actions. External
enforcement ensures a bad plan cannot cross the boundary. Systems need both
because many governance rules guide choices before a final call, while some
risks permit a hard execution-time check.

The distinction also clarifies which rules belong in context. A secret-access
policy enforced entirely by the filesystem still benefits from explanation in
the agent's prompt, though the filesystem owns security. A preference such as
"use the EU reporting channel" may shape several earlier decisions and deserves
protected carriage even if the last API performs a region check.

## Recovery beats permanent deletion

Keeping every byte of a large tool observation active is expensive, while
summarizing it destroys exact strings. Recent work on addressable recall offers
a third option: move old observations to an
append-only store, leave a compact citation in context and let the agent request
the original by ID.

Addressable Recall Compaction reported 99.40 percent exact-answer accuracy on
its needle-in-a-haystack evaluation, against 88.12 percent for the strongest
baseline in that experiment. Its LongBench-v2 Hard gain was smaller, 29.97
versus 28.25 percent. These are benchmark results from a preprint, though the
mechanism is useful independent of the headline: eviction can remain reversible.

This produces a practical hierarchy. Authority stays pinned verbatim. Bulky
exact evidence can live behind a citation, while working state is safer in a
schema. Deletion is reserved for material with no plausible recovery value.

## Test forgetting as a first-class behavior

Most agent evaluations stop when the first task succeeds. Compaction needs a
longer test. The same constraint should be exercised before and after every
memory transition.

Build deterministic scenarios around deployment-specific rules. Trigger enough
benign work to force compaction. Then ask for an action that would have been
allowed without the rule. Inspect the tool arguments, not the explanation. Run
the test across repeated compactions as well: ConstraintRot reports 78 percent
violation after four rounds, versus zero before compaction.

Add adversarial content to retrieved documents and tool results. Ask the
compactor to treat the policy as obsolete, irrelevant or too expensive to keep.
Integrity checks should reject every candidate that loses protected state.

Finally, test the system when compaction itself fails. Invalid structured
output, an unavailable summarizer or an inflated summary should preserve the
old history or stop safely. Memory pressure must never silently weaken
authority.

Long context made extended agent work possible. Compaction keeps it running.
That makes the compactor part of the control plane, even when its interface
looks like text cleanup. Govern how the agent forgets.

## Sources

1. Chen, [Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents](https://arxiv.org/abs/2606.22528), arXiv preprint, 2026.
2. Sharma et al., [State Compression in Two-Agent LLM Relays](https://arxiv.org/abs/2607.18265), arXiv preprint, 2026.
3. Dang et al., [Addressable Recall Compaction for Long Context-Window Control in AI Agents](https://arxiv.org/abs/2607.25066), arXiv preprint, 2026.
4. Chen et al., [Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents](https://arxiv.org/abs/2605.08580), arXiv preprint, 2026.
5. Liu et al., [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172), TACL 2024.