Step by Step
N
No persistent memory — everything must be in context
LLMs have no persistent memory between separate conversations — everything relevant must be included within the current context window for the model to have access to it.
Example: a model having no memory of a conversation from yesterday unless that conversation's content is explicitly re-included in today's context window.
T
Token limits vary significantly by model
Different models offer significantly different context window sizes: GPT-4o offers 128K tokens, Claude 3.5 offers 200K tokens, and Gemini 1.5 Pro offers 1M tokens (enough for an entire codebase).
Example: choosing Gemini 1.5 Pro specifically because a task requires including an entire large codebase (needing close to 1M tokens) that wouldn't fit within GPT-4o's 128K token limit.
L
Lost in the middle — attention degrades for buried content
LLMs pay measurably less attention to important information buried in the middle of a long context, compared to information placed at the very start or end — meaning critical content should be positioned at the beginning or end of a long prompt whenever possible.
Example: placing the most critical instructions or facts at the very beginning or very end of a long prompt, rather than burying them in the middle, to maximize the chance the model actually attends to them.
M
Memory strategies — RAG, summarization, sliding window
Since context windows have limits and attention degrades for buried content, strategies like RAG (retrieving only the most relevant chunks), summarization (condensing older context), and sliding windows (keeping only recent context) all help manage the practical limitations of context windows.
Example: using RAG to retrieve only the most relevant document chunks for a specific query, rather than trying to stuff an entire massive document collection into the context window at once.
Applied Walkthrough
1
A developer includes a very long document in a prompt, with the most critical instruction buried somewhere in the middle of that long document.
2
Ask: is the model likely to pay full attention to that critical middle-buried instruction? Not necessarily — the "lost in the middle" phenomenon means LLMs pay measurably less attention to content buried in the middle of long contexts.
3
The developer restructures the prompt, moving the critical instruction to either the very beginning or very end of the prompt instead.
4
This restructuring significantly increases the likelihood the model actually attends to and follows that critical instruction, directly addressing the lost-in-the-middle limitation.
Exam Application
Exams test whether you know the specific context window sizes for major models (GPT-4o: 128K, Claude 3.5: 200K, Gemini 1.5 Pro: 1M) and whether you understand the lost-in-the-middle phenomenon and its practical implication: place critical content at the start or end of a long prompt, not buried in the middle.
⚠ Common Trap
The most common trap is assuming a larger context window eliminates the need to think carefully about where critical information is placed within a prompt. Even with a very large context window, the lost-in-the-middle phenomenon still applies — critical content buried in the middle of a long context can still receive less attention than content placed at the start or end.
✓ Quick Self-Check
1. Do LLMs have persistent memory between separate conversations?
No — everything relevant must be included within the current context window.
Tap to reveal / hide
2. What are the approximate context window sizes for GPT-4o, Claude 3.5, and Gemini 1.5 Pro?
GPT-4o: 128K tokens; Claude 3.5: 200K tokens; Gemini 1.5 Pro: 1M tokens.
Tap to reveal / hide
3. What is the "lost in the middle" phenomenon?
LLMs pay measurably less attention to important information buried in the middle of a long context, compared to the start or end.
Tap to reveal / hide
4. Where should critical content be placed in a long prompt, based on this phenomenon?
At the very start or very end of the prompt, not buried in the middle.
Tap to reveal / hide
5. Name one strategy for managing context window limitations.
RAG, summarization, or sliding window (any one).
Tap to reveal / hide