More compact, not less truthful

Context Compression

Compression is not just making text shorter. It is preserving the facts, caveats, priorities, and provenance needed for the current task while removing material that does not change the answer.

Key facts

Context Compression

  • Compression should happen after relevance selection, not before it.
  • Safe compression preserves answer-critical facts, qualifiers, conflict notes, and source provenance.
  • Extraction, abstraction, structural compression, and memory tiering solve different token-budget problems.
  • Evaluate compression with answer quality and evidence preservation, not token reduction alone.

Compression is useful because context windows are expensive and imperfect. Long prompts cost more, add latency, and can bury the facts the model needs. But careless compression creates a different failure: the prompt is short, clean, and wrong. A good compression strategy has a loss budget. It names what may be dropped, what must be preserved, and how the loss will be detected.

"up to 20x compression with little performance loss"
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

Compression starts with the query

The same document compresses differently for different tasks. A security audit, a refund policy answer, and a release-note summary need different spans from the same source. This is why question-aware compression is usually safer than generic summarization. It can preserve answer-bearing facts and discard background that is irrelevant to the current query.

LongLLMLingua extends the prompt-compression idea for long-context scenarios by estimating document-level relevance before fine-grained compression. The lesson for practitioners is simple: compressing every chunk equally is rarely optimal. Dense evidence should keep more budget than low-signal framing text.

Four compression modes

Extraction

Extraction keeps exact spans and removes surrounding text. It is best for policies, contracts, code, logs, and anything where wording matters. Use it when the model must cite or reason from precise language. The risk is orphaning a sentence from its qualifier, such as an exception, date range, or jurisdiction.

Abstraction

Abstraction rewrites content into a summary. It is best for meeting notes, long discussions, and narrative material where exact wording matters less than decision state. The risk is introducing claims not present in the source or smoothing over disagreement.

Structural compression

Structural compression changes shape rather than meaning: tables, bullet claims, source cards, timelines, contradiction lists, and decision records. It often beats prose summaries because each field has a purpose and can be checked.

Memory tiering

Memory tiering keeps the active window small by moving stable facts and historical detail into slower storage. MemGPT frames this as virtual context management: the system can retrieve and update memory tiers instead of treating the prompt as the only place information can live.

Chunking is not compression by itself

Chunking divides a source into retrievable units. It reduces what enters the prompt, but it can also destroy meaning when boundaries split a rule from an exception or a claim from its source. Good chunking is semantic: headings, tables, code symbols, timestamps, and policy sections are better boundaries than fixed token counts alone.

For high-stakes answers, include parent metadata: document title, section path, date, owner, and a link to the full source. A compressed chunk without provenance is just a plausible fragment.

How to verify compression

A compression pipeline needs regression tests. Build a small set of questions whose answers depend on facts that are easy to drop: a negative condition, a threshold, a date, a nested exception, a minority opinion, or a code path with similar names. Run the model with the original context and the compressed context. The compressed version must preserve the answer and cite the same decisive evidence.

Track three metrics together: token reduction, answer quality, and evidence preservation. Token reduction alone rewards reckless summarization. Answer quality alone can hide lucky guesses. Evidence preservation checks whether the model still has the facts it needs.

Sources Used

Cite This Page

Curated Context. "Context Compression." Accessed July 6, 2026. https://curatedcontext.com/compression

https://curatedcontext.com/compression