Why AI Forgets Your Input and How to Fix It

What Is a Context Window?

Every model has a context window , the total number of tokens it can “remember” at one time. Both the input you provide and the model’s output must fit inside this window.

Think of it as a fixed‑size table: the model must place the question, any pasted document, conversation history, and system instructions all on the table simultaneously.

Older models may have windows of about 4,000 tokens (≈3,000 words, six pages). Newer models can handle 128,000 tokens or even over a million tokens, comparable to a short novel or an entire codebase. However, a larger window does not guarantee equal attention to every part of the content.

Two Forms of the Problem

Document Scenario

If you paste a twenty‑page legal contract and ask about section 7, the model might find the correct answer, might miss it, or might answer from the wrong section. The more surrounding text, the more the model’s attention is diluted, even if the window isn’t full.

Conversation Scenario

By default the model has no persistent memory. Each new user message causes the model to reread the entire conversation from the first turn to the latest input, increasing token usage each round.

A typical exchange consumes about 50 tokens for the prompt and 300 tokens for the answer, totaling 350 tokens per turn. Ten turns ≈ 3,500 tokens; twenty turns ≈ 7,000 tokens. A long afternoon of back‑and‑forth can easily reach 20‑30 k tokens, and each token is both a memory unit and a billing unit.

The “Lost in the Middle” Phenomenon

When the input is long, the model focuses most on the beginning and the end, while the middle content is often ignored. Researchers call this “lost in the middle.” It explains why early conversation details fade as the dialogue grows: early messages drift into the middle of the window, the area of weakest attention.

What You Can Do

1. Use a Model with a Bigger Window

Switching to a model with a larger context window gives you more “space,” but it does not automatically make the needed content easier to find, so the following tactics remain important.

2. Chunk the Input

Only provide the portion that is relevant. If you need information from chapter 3, feed just that chapter instead of the whole document, reducing noise and strengthening the signal.

3. Summarize Before Asking

First ask the model to summarize the document, then pose your actual question on the summary. This uses two calls but gives a focused context for the second call.

4. Position Key Information

Place critical details at the very start or very end of the prompt. When writing prompts, put the main question at the end or the most important background at the beginning, avoiding burying it in the middle.

5. Restate Important Constraints

If a crucial constraint was mentioned in the first message, repeat it in later turns (e.g., in turn 15) to keep it in view, even if it costs a few extra tokens.

6. Leverage System Prompts

Many platforms let you set persistent system instructions (e.g., “custom instructions” in ChatGPT, “system prompt” in Claude, or the system‑prompt field in Amazon Bedrock). Use clear language for stable rules, but still restate key directives in the current message.

7. Start a New Conversation When Needed

If the dialogue has drifted after many turns, begin a fresh conversation and carry over only the essential context.

8. Build Your Own Memory Layer

Summarize earlier turns into a short excerpt and store it (in a database, file, or variable). Inject this summary at the start of each new call, effectively creating a cache for the conversation context.

For developers, this mirrors using Redis in front of Postgres: cache expensive repeated work and only send new content each time.

Some platforms also offer prompt caching , where system prompts or repeated context are processed once and reused across calls, saving token cost.

In document‑heavy use cases, retrieval‑augmented generation (RAG) is preferable: retrieve the most relevant fragments instead of feeding the entire document.

The core principle is: give the model less text, but give it the right text.

Key Takeaways

If you’re just starting out: Understand that the context window limits how much the model can attend to. Ask about specific sections, restate important points in long dialogues, and start a new conversation when the model seems to drift.

If you’re a developer: Treat the context window size as a specification, not a performance guarantee. A million‑token window does not equal perfect memory. Place critical information at the edges, and implement summarization or caching for early conversation turns.

Positional bias and the “lost in the middle” effect mean that indiscriminately adding more tokens weakens attention and drives up infrastructure costs—a “prose tax.” The real fix lies in stricter ingestion boundaries, treating all incoming strings as untrusted telemetry, applying a “sieve‑and‑sign” pattern to strip noise, and sending a concise, deterministic state schema to the model.