Skip to main content

Retrieval-Augmented Generation (RAG)

Intermediate
What you'll learn
  • What RAG is and the retrieve-augment-generate loop
  • How to index, retrieve, augment, and generate with citations
  • Why RAG beats fine-tuning for 'answer about my docs' needs
  • The five failure modes that kill RAG quality
  • A copy-paste grounding prompt that closes the two biggest gaps

RAG makes a model answer questions about your data — docs, a knowledge base, a codebase — that it was never trained on. The idea is simple: retrieve the relevant pieces, augment the prompt with them, then generate an answer grounded in those pieces.

The loop​

Guided walkthrough1 of 4
  1. Split into chunks, embed them (see /docs/foundations/embeddings), and store in a vector (and/or keyword) index.

For the embedding step in indexing, see Embeddings & Vector Search.

Why RAG instead of fine-tuning?​

Pro tip
  • Fresh: update the data, not the model
  • Verifiable: provides citations
  • Cheap: far cheaper than retraining

For most "answer about my documents" needs, RAG is the right first tool — see Fine-tuning vs Prompting vs RAG.

The failure modes (where RAG quality dies)​

Watch out
  • Bad retrieval = bad answer. If the right chunk isn't retrieved, the model can't use it. Most 'RAG is wrong' problems are retrieval problems.
  • Chunking too coarse/fine wrecks relevance (see embeddings).
  • No grounding instruction: the model blends retrieved facts with its own guesses. Tell it to answer only from context and to admit gaps.
  • Stuffing too much: irrelevant chunks dilute the signal and cost tokens. Retrieve few, high-quality chunks.
  • No citations: you can't verify, so you can't trust.

The chunking failure ties back to embeddings, and over-stuffing costs tokens.

Pro tip
  • Evaluate retrieval separately: measure 'did we retrieve the right chunk?' apart from 'did the model answer well?' It localizes the problem fast. See Evals (/docs/foundations/evals).

Copy-paste: a grounding prompt​

The single highest-leverage fix is a grounding instruction. Drop your retrieved chunks into a template like this — it forces the model to answer only from context, cite each claim, and admit gaps instead of guessing:

Grounding prompt

You are answering strictly from the context below.

Rules:
- Use ONLY the context to answer. Do not use outside knowledge.
- Cite the source after each claim, like [chunk 2].
- If the answer is not in the context, reply exactly:
"I don't have that in the provided sources."
- Quote numbers and names verbatim — never paraphrase a figure.

Context:
[chunk 1] ...
[chunk 2] ...
[chunk 3] ...

Question: <the user's question>

Pair it with a few high-quality chunks (not everything you retrieved) and you close the two biggest gaps at once: hallucinated blending and unverifiable answers. Then eval retrieval and generation separately so you know which half to tune.

Master the terms​

Press Enter or Space to flip the card. Use the left and right arrow keys to move between cards.Term shown.
1 / 6

Check yourself

0/5
  1. What do the three letters in RAG stand for, in order?
  2. When 'RAG is wrong', what is most often the real problem?
  3. Why is RAG usually preferred over fine-tuning for 'answer about my documents'?
  4. What is the single highest-leverage fix to stop the model blending facts with guesses?
  5. Why evaluate retrieval separately from generation?
Key takeaways
  • RAG = retrieve relevant chunks, augment the prompt, generate a grounded, cited answer.
  • Index (chunk + embed + store), retrieve top chunks, augment with a grounding instruction, generate with citations.
  • Prefer RAG over fine-tuning for document Q&A: fresh, cited, cheaper.
  • Most failures are retrieval failures — retrieve few high-quality chunks, not everything.
  • Always add a grounding instruction and cite; eval retrieval and generation separately.

Next​