Selection before structure

What Belongs in the Context Window

The context window is a working set, not a storage bucket. Include the smallest set of information that lets the model do the current job accurately, traceably, and within the rules.

Key facts

What Belongs in the Context Window

  • The context window is a working set for the current task, not a storage bucket.
  • Answer-critical task details, priority instructions, authoritative evidence, necessary examples, durable memory, and the output contract are the usual inclusion layers.
  • Raw history, stale summaries, unlabeled snippets, and duplicate source dumps usually belong in retrieval or citations instead of the active prompt.
  • Similarity search is only a candidate generator; curate by expected answer impact, source authority, freshness, and evaluability.

Good context is task-shaped. A support bot answering "what changed in my invoice" needs the current invoice, the account's billing rules, and the user's recent plan change. It does not need the whole CRM history. A coding agent fixing one failing test needs the failure, the local contract, nearby implementation, and repo conventions. It does not need every issue ever filed against the component.

"transcends simple prompt design"
A Survey of Context Engineering for Large Language Models - on context engineering as a broader discipline

The inclusion test

Before adding any token, ask five questions. First, does it directly change the answer or the work plan? Second, is it more authoritative than another source already included? Third, is it current enough for the decision? Fourth, can the model tell what role it plays? Fifth, can you evaluate whether it helped? If the answer is no, the content probably belongs in storage, retrieval, or a linked source list rather than the active prompt.

What usually belongs

The first layer is the job itself: the user request, success criteria, assumptions, and any explicit non-goals. This layer keeps the model from optimizing for a nearby but wrong task. The second layer is instruction context: policies, formatting rules, tool limits, safety boundaries, and local conventions. These are not "facts" in the ordinary sense, but they are context because they shape valid behavior.

The third layer is evidence. Evidence includes retrieved passages, database records, logs, code snippets, user-provided files, and source metadata. The best evidence is not merely semantically similar to the query; it is specific enough to answer the question and carries provenance so conflicts can be resolved. Retrieval-augmented generation was introduced to give models access to explicit non-parametric memory rather than relying only on stored model weights, and that idea still matters even with larger context windows.

The fourth layer is examples. Examples belong when format, style, edge cases, or domain patterns are easier to demonstrate than describe. They become harmful when they silently introduce outdated facts, override the current task, or crowd out direct evidence.

What usually stays out

Raw history is the most common source of context rot. Conversation transcripts, issue threads, full documents, or search result dumps often contain repeated, stale, contradictory, and low-authority material. Keep the durable decision, the current constraint, and the source link; leave the rest retrievable. If a detail is only useful for audit, it belongs in a citation or trace, not necessarily in the active reasoning window.

Also keep out hidden equivalences. A document title, a Slack message, and a database row do not have the same authority. If they are all included without labels, the model has to infer which one wins. Context curation should remove that burden by labeling source type, timestamp, owner, and confidence.

Signal beats similarity

Similarity search is a candidate generator, not a context policy. A passage can be close to the query and still be useless because it is generic, stale, or at the wrong level of detail. Conversely, a short configuration line, product rule, or prior decision can be decisive even if it is not linguistically close to the user's wording. Curated context ranks by expected answer impact, not by embedding score alone.

This is why production RAG systems need operational validation. Barnett and coauthors describe failure modes such as missing content, missed top-ranked documents, and incorrect specificity. Those are selection failures: the answer goes wrong before generation starts.

A practical context inventory

Layer Include Exclude
Task User goal, acceptance criteria, current question, known constraints. Adjacent goals, speculative future work, unrelated history.
Instructions Priority rules, output contract, tool and policy boundaries. Duplicated rules, examples that conflict with higher-priority instructions.
Evidence Authoritative passages, fresh records, exact code/log snippets, source metadata. Full dumps, stale summaries, unlabeled snippets, low-authority duplicates.
Memory Stable preferences, durable decisions, user-approved facts. Every past turn, inferred preferences, unresolved assumptions.

The rule of thumb

Put the answer-critical material in the prompt. Put the audit trail in citations. Put the rest behind retrieval. A context window should read like a brief for the current decision, not a warehouse manifest.

Sources Used

Cite This Page

Curated Context. "What Belongs in the Context Window." Accessed July 6, 2026. https://curatedcontext.com/what-belongs-in-context

https://curatedcontext.com/what-belongs-in-context