Context Windows: What Fits, and What Degrades

The advertised window is a ceiling, not a working budget

The distinction matters because the two numbers are used for different decisions and get confused constantly. The ceiling tells you whether a request will be rejected. Your effective context tells you whether the answer will be right. A system that stays under the ceiling and above the effective limit does not fail loudly, it just becomes unreliable in a way that looks like a prompt problem.

Providers change these limits regularly, so the number for your model belongs in your model registry rather than in an article. Read it from the provider's current documentation, record it beside your model pin, and derive every cap in this guide as a fraction of whatever that number is.

The vocabulary

Context window

The maximum number of tokens a model can process in a single request, counting the system prompt, tool definitions, conversation history, retrieved material, the user's message and the space reserved for the response.

Effective context

The amount of material a model actually uses reliably for a given task, measured by placing a known fact at different positions in contexts of increasing length and recording where accuracy falls. It is a property of your model, your task and your data, not a published figure.

Output reservation

The tokens set aside for the response, which on most APIs are subtracted from the same window as the input, so a large reservation directly reduces how much evidence you can send.

Prefix cache

A provider-side store of the computed state for a prompt prefix, reused when a later request begins with the identical token sequence, which reduces the cost and latency of the repeated portion.

Compaction

Replacing a growing conversation history with a shorter representation that preserves what later turns depend on, either as a summary, a structured state object, or a searchable store of past turns.

What is actually consuming the window

Six things share the budget, and teams usually track only two of them. The ones that get missed are the ones that grow without anybody deciding they should.

The tool row deserves more attention than it gets. Tool definitions are part of the prompt, so twenty tools with carefully written descriptions is a permanent per-call cost on every request, including the ones that need none of them. It also makes selection harder, since the model must choose from a longer menu. If your agent has accumulated tools, both the cost and the accuracy argument point the same way, and designing tools for agents covers how to consolidate them.

Why quality falls before the limit does

Three mechanisms degrade long-context performance, and they are separate problems with separate fixes. Naming them individually is what stops a team from responding to all three by shortening the prompt and hoping.

Attention is spread across everything you sent

A model attends over the whole sequence, so every additional token is another candidate for the model's attention when it looks for the one that matters. Doubling the context does not halve the accuracy, but it does dilute the signal, and the effect is strongest for a single small fact buried inside a large body of related text. This is why a needle in an otherwise empty haystack is an easy test and a poor proxy for real work.

Position is not neutral

Material at the start and end of a long context is generally used more reliably than material in the middle. The size and shape of that effect varies by model and changes between releases, so treat it as a property to measure rather than a constant to design around. The practical consequence is stable: if one passage matters more than the others, do not leave its position to the order your retriever happened to return.

Distractors are worse than length

This is the mechanism that surprises people. Filling a context with passages that are topically similar to the answer, which is exactly what a retriever produces, is harder for a model than filling it with unrelated text. The near misses compete. A retriever tuned for recall by raising k is therefore buying coverage with interference, and past a certain point it trades away more accuracy than it gains. That point is specific to your corpus and it is worth finding, because the fix is a reranker and a lower k rather than a larger window.

Published figures on any of this describe someone else's model, task and corpus, and they move with every model release. The section on measuring your own effective context gives you a number you can defend, and it takes an afternoon.

Allocate the window before anything consumes it

The default behaviour of a context window is first come, first served: whatever the assembly code appends first gets the space, and whatever runs last gets truncated by a library nobody remembers configuring. That is a design decision made by accident. Replace it with an explicit allocation.

# context/assembly.yaml
# Caps are FRACTIONS of whatever window the pinned model has, never raw token counts.
# Change the model and the mix stays proportional instead of silently rearranging itself.

window_source: model_pin          # read from the model registry at build time, never hardcoded
safety_margin: 0.05               # never allocated, absorbs tokenizer disagreement

envelopes:

  instruction:
    cap: 0.06
    owner: "@rina"
    overflow: fail_build           # a longer system prompt is a design decision, not an accident
    position: 1                    # first, and byte-identical across calls

  tools:
    cap: 0.08
    owner: "@platform"
    overflow: fail_build           # adding a tool means removing one or raising the cap on purpose
    position: 2
    note: "serialised schemas are re-sent on every call. 20 tools is a permanent tax."

  evidence:                        # retrieved passages
    cap: 0.34
    position: 3
    k: derived                     # k = floor(cap_tokens / mean_chunk_tokens), never a magic number
    overflow: drop_lowest_score
    require_citation_ids: true

  history:
    cap: 0.32
    position: 4                    # after the stable prefix, because it changes every turn
    eviction: structured_state     # see history.yaml, not a rolling prose summary
    always_keep:
      - "the pinned state object"
      - "the last 4 turns verbatim"

  reserve:                         # output plus headroom, never borrowed from
    cap: 0.15
    min_output_tokens: 800
    on_breach: shrink_evidence_first    # explicit precedence, decided once, not at runtime

cache:
  stable_prefix: [instruction, tools]
  forbidden_in_prefix: ["current_datetime", "user_id", "session_id", "request_id", "greeting"]
  assert_prefix_byte_identical: true

build_assertions:
  - "sum(envelope caps) + safety_margin <= 1.0"
  - "every envelope has an overflow policy"
  - "token counts come from the provider tokenizer, not from a character estimate"
  - "assembled prompt is counted and logged per envelope on every call"

Compacting history without losing the constraints

History is the envelope that overflows first in any conversational system, and the strategy you pick decides which information survives. All four options below are in production somewhere. Only one of them keeps a constraint stated in turn three alive at turn ninety.

The failure mode of the rolling summary is worth stating plainly, because it is the most common design and it degrades in a way that produces no error. Each summarisation is lossy. Summarising a summary compounds that loss, and the constraint the user stated once in turn three, no weekend deliveries, disappears somewhere around the fourth compaction with nothing in any log to say it went. The model then produces a confident plan that violates it, and the transcript shows the user stating it clearly, which makes the bug look like the model ignoring an instruction rather than never receiving one.

The structured state pattern fixes this by separating two things that a summary conflates. The conversation is compressible. The facts stated in it are not. Extract facts, decisions, constraints and open questions into a schema, carry that object verbatim on every turn, and keep only the last few turns as prose. Constraints are marked as never evicted, whatever the cap says. The state object is inspectable, diffable between turns, and testable, which a prose summary is not. This connects directly to agent memory design, where the same object usually becomes the durable memory.

// context/history_state.schema.json
// Conversation history compacts into THIS, not into a prose summary.
// Prose summaries of prose summaries lose a constraint per generation and never say which one.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["facts", "decisions", "constraints", "open_questions", "turn_range"],
  "properties": {

    "facts": {
      "description": "Stated by the user, never inferred. Each carries the turn it came from.",
      "type": "array", "maxItems": 40,
      "items": {
        "type": "object",
        "required": ["key", "value", "turn"],
        "properties": {
          "key":   { "type": "string" },
          "value": { "type": "string", "maxLength": 200 },
          "turn":  { "type": "integer" }
        }
      }
    },

    "decisions": {
      "description": "Choices already made. Re-opening one requires the user to say so.",
      "type": "array", "maxItems": 20,
      "items": { "type": "string", "maxLength": 200 }
    },

    "constraints": {
      "description": "Hard limits: budget, dates, exclusions, compliance. NEVER evicted, whatever the cap.",
      "type": "array", "maxItems": 20,
      "items": { "type": "string", "maxLength": 200 }
    },

    "open_questions": {
      "type": "array", "maxItems": 10,
      "items": { "type": "string", "maxLength": 200 }
    },

    "turn_range": {
      "description": "Which turns this state covers, so the verbatim tail is not double counted.",
      "type": "object",
      "required": ["from", "to"],
      "properties": { "from": { "type": "integer" }, "to": { "type": "integer" } }
    }
  }
}

Prefix caching changes both the arithmetic and the layout

Providers can cache the computed state of a prompt prefix and reuse it when a later request begins with the identical token sequence, which reduces the cost and the time to first token for the repeated part. The mechanism is exact prefix matching, and that single fact dictates how you order the prompt.

• Stable content first. System prompt, then tool schemas, then any long document shared across requests. Volatile content goes last: the retrieved passages, the history, the user's turn.
• Nothing volatile above the line. A current timestamp, a user id, a session id, a request id or a personalised greeting placed near the top invalidates the cache for every request behind it. This is the most common cache bug and it is invisible in behaviour, since the output is identical and only the bill and the latency move.
• Byte-identical means byte-identical. A reformatted system prompt, a changed indent, a trailing newline added by a linter, all break the match. Keep prompt text in its own file and assert on the prefix in continuous integration, as covered in prompts as code.
• Check the provider's rules rather than assuming. Minimum prefix length, cache lifetime, whether caching is automatic or requested explicitly, and how a cached read is priced all vary by provider and change. Read the current documentation and record what you found beside your model pin.

The consequence for design is that a long stable prefix is cheaper than its token count suggests, and a short prompt with a volatile header is more expensive than its token count suggests. That inverts some architectural instincts. Sending the same reference document on every call can be reasonable if it sits in a cached prefix, while a carefully minimised prompt that begins with the current date is paying full price on every request.

Long context or retrieval

This is not a choice between two architectures so much as a choice about where the selection happens. Retrieval selects before the call, long context selects during it. The trade-offs are stable even as window sizes grow.

In practice most production systems use both: retrieval to select the evidence, and a generous window so the selected evidence arrives whole rather than clipped. The case for stuffing is strongest when the corpus is small, stable and shared across requests, because prefix caching then makes the token count much cheaper than it looks. The case for retrieval is strongest when the corpus is large, changes often, or when you need to show a user which passage an answer came from. RAG explained for builders covers the retrieval side in full.

Measure your own effective context

Every number published about long-context degradation was measured on a particular model, task and corpus, and it moves with each model release. Yours takes an afternoon to measure and it is the only one you can defend in a design review.

1. Collect thirty to fifty real questions with known answershalf a day

   From your own corpus, with the passage containing each answer identified. Use questions people actually ask, including the ones with a near-duplicate elsewhere in the corpus, since those are the cases the probe is really testing.

2. Build contexts at increasing lengthsscripted

   For each question, assemble contexts at several sizes across the range you might realistically use, up to the ceiling. Keep the question and the target passage constant so length is the only variable that moves.

3. Fill with your own distractors, never with fillerimportant

   Pad with the passages your retriever actually returns for that query, not with unrelated text. Random filler makes the probe far easier than production and produces a flattering number that will not hold. The similarity of the distractors is most of what you are measuring.

4. Vary the position of the target passagescripted

   Place it near the start, in the middle and near the end of each context length. This separates a length problem from a position problem, and the two have different fixes: shorter contexts against reranking and deliberate ordering.

5. Score, and read the transitionsan hour

   Plot accuracy against length for each position. The useful output is the length at which the middle position starts falling, since that is your working ceiling for anything that matters. Read the failures rather than only counting them, because the shape of a wrong answer tells you whether the model missed the passage or found a distractor.

6. Re-run it on every model changescripted, cheap

   This is why you script it. A new model build, a different provider or a change to your chunking all move the curve, and the probe is the only way to find out before your users do.

Two things usually come out of the first run. The working ceiling is lower than the advertised window by a wide margin, and the middle position falls first. Both are useful in a way a vendor number is not, because you can point at your own chart when someone proposes raising k to fix a recall complaint.

Fit and cost, on your own numbers

Once the envelopes are named, the arithmetic is straightforward and worth running before the design rather than after the first oversized request. Enter your pinned model's window and your measured token counts.

The maximum chunks output is there to make a point rather than to be used. It shows what the window would permit, which is usually many times more than the number that produces good answers, and the gap between those two figures is the whole subject of this guide. Set k from your evidence envelope and your probe results, not from what fits. The general cost model sits in token cost arithmetic.

That review takes about an hour and it removes the two failures that account for most context incidents: a request that grew past the ceiling because nothing capped it, and a quality drop from evidence that was expanded on the assumption that more is better. Both are budget failures rather than model failures, which is why the fix is a configuration file. This is the assembly layer we build into every custom LLM application rather than something added after the first oversized request.