Prompting for Long Context
- Where to put long documents in a prompt (top, not bottom) and why it matters up to ~30%
- How to structure multiple documents with <document> / <source> tags
- The quote-extraction trick that makes long-doc answers dramatically more grounded
- When 1M context is the wrong tool — and retrieval or chunking is right
- How context awareness and compaction change the game on Sonnet 5
Long context changes the game — and it can also silently ruin your answers. Since Sonnet 5, Opus 5, and Opus 4.6/4.7 ship 1M-token windows as the default (no beta header), the temptation is to paste everything and hope. Don't. The whole difference between a great long-context prompt and a slop one is structure — where each part goes and how you label it.
The four techniques that actually move the needle
- Place long documents and reference material above your instructions and query, not below. Anthropic's own tests show queries at the end can improve response quality by up to 30% — especially with complex, multi-document inputs. This is the single highest-leverage change.
- For multi-doc inputs, wrap each in <document index='N'> containing <source>filename</source> and <document_content>…</document_content>. Metadata lets Claude cite which doc it used and stops it from cross-contaminating claims.
- Instruct Claude to pull the relevant passages from each document into <quotes> tags before answering. This forces attention on the parts that actually matter and drops the noise. Answers become traceable to source lines.
- After a long block of context, one-line restate the ask (see also: prompting/basics — 'Burying the ask' anti-pattern). First and last positions carry the most weight.
The canonical shape
Anthropic's own template — copy this and you're already ahead of most people using 1M windows:
Long-context multi-document template
<documents>
<document index="1">
<source>annual_report_2025.pdf</source>
<document_content>
{{ANNUAL_REPORT}}
</document_content>
</document>
<document index="2">
<source>competitor_analysis_q2.xlsx</source>
<document_content>
{{COMPETITOR_ANALYSIS}}
</document_content>
</document>
</documents>
Find quotes from the two documents that are relevant to identifying strategic advantages we can press on in Q3. Place them in <quotes> tags with the source filename. Then, based only on those quotes, recommend three focus areas with a one-line justification each. Place your recommendations in <recommendations> tags.Notice the three moves this prompt makes at once:
- Docs first, question last — the actual ask lives below the material.
- Named sources — every doc has a filename Claude can cite.
- Quotes → answer — Claude has to ground itself before recommending anything.
Context rot: why more tokens ≠ better answers
More context is not automatically better. As token count grows, accuracy and recall degrade — Anthropic calls this context rot. Models tend to use the beginning and end of a long input more reliably than the middle (the classic "lost in the middle" effect).
Practical consequences:
- Trim before you paste. A 100k-token dump with 20% irrelevant sections often loses to the 60k-token curated version.
- Order for salience. Put the doc most likely to contain the answer first inside
<documents>. - Prompt caching pays for itself fast. Stable-prefix design (system prompt + docs identical across turns) means the cached prefix reads at ~10% of the token price. See Prompt Caching.
- 1M window ≠ 1M tokens of useful attention. Accuracy degrades as you fill the desk.
- Bury the ask in the middle of a huge paste and it will get under-weighted.
- Above 200k input tokens on Sonnet 5 / Opus 5, you're on long-context pricing — check the model page before pasting a repo.
When 1M context is the WRONG tool
Long context is tempting because "just paste the codebase" feels simpler than building retrieval. Sometimes it is. Often it isn't. Choose the alternative when:
| Signal | Better than 1M paste |
|---|---|
| You need the same doc across thousands of queries | Retrieval + RAG — cheaper, faster |
| Users bring their own documents each turn | Files API with document uploads |
| The answer requires the whole codebase but you re-ask often | Prompt caching on a stable-prefix codebase snapshot |
| You need auditable citations at line granularity | Retrieval with chunk IDs → cite chunk, not "somewhere in the paste" |
| Latency matters more than one-shot quality | Chunk + rerank, then only pass the top-k |
Rule of thumb: if you'd hesitate to re-paste this every turn, you probably want retrieval instead.
What changes with Sonnet 5 (context awareness)
Sonnet 5, Sonnet 4.6, Sonnet 4.5, and Haiku 4.5 now have context awareness — the API injects a running budget into the system prompt so the model can pace itself:
<budget:token_budget>1000000</budget:token_budget>
After each tool call, the API updates it:
<system_warning>Token usage: 350000/1000000; 650000 remaining</system_warning>
You don't send these tags — the API does. Practical effect: on long agentic runs, Sonnet 5 will proactively summarize or hand off before hitting the wall, instead of guessing. Opus 4.7+, Fable 5, and Mythos 5 don't get injected tags; give them explicit budgets via task budgets (beta) instead.
Common mistakes
- Pasting a doc after the instruction — the single most common miss. Move it above the ask.
- No
<source>metadata — Claude cites "the document" instead ofreport.pdf; downstream tools can't verify. - One giant
<document>blob — collapse multiple sources into one and Claude can't tell them apart; use one per file. - Asking for the answer directly on long inputs — always request quotes first for anything above ~20k input tokens.
- Assuming caching is automatic — it isn't. See Prompt Caching for cache breakpoints and TTLs.
- Forgetting compaction exists — for very long agent runs on Claude 4.6+, server-side compaction summarizes older turns automatically.
Try it now
Take a 20k+-token doc you'd normally paste raw. Wrap it in the template above, put your ask at the bottom, and add "Extract relevant quotes first" as the first instruction. Compare the answer to your old prompt — the difference is usually not subtle.
Check yourself
0/5- Longform data at the top; the ask at the bottom — up to ~30% better on complex inputs.
- Wrap every doc in <document> with <source> metadata; ask for <quotes> before the answer.
- 1M context ≠ free — context rot is real, and long-context pricing kicks in above 200k input tokens.
- Use RAG when the same corpus serves many queries; use compaction and prompt caching to make long context economical.
Next
- Tokens, Context & Memory — the mental model behind the desk
- Prompt Caching — how to make long-context economics work
- Memory & Context Editing — server-side compaction and clearing on Claude 4.6+
- Retrieval-Augmented Generation — when to reach for RAG instead
- XML Tags for Structure — the tagging habit that makes this all work