Skip to main content

Fine-tuning vs Prompting vs RAG

Intermediate
What you'll learn
  • The three levers you have when the model isn't doing what you want
  • The order that actually works — and why fine-tuning is almost never step 1
  • When RAG is the right fix and when it's overkill
  • What fine-tuning is genuinely good at (and what it isn't)
  • How the three combine in a real production system

When the model doesn't do what you want, there are three levers — and people reach for the expensive one first. Here's the order that actually works.

Try in this order​

Guided walkthrough1 of 3
  1. Clearer instructions, examples, a role, output constraints (see /docs/prompting/basics). Fixes the majority of problems, costs nothing extra, and is instant to iterate. Most 'the model is bad at X' turns out to be 'the prompt was vague.'

The decision table​

Your problemReach for
Vague/wrong outputs, wrong formatPrompting
Doesn't know your data / needs current infoRAG
Needs a very specific style/behavior, consistently, at scaleFine-tuning
Needs to take actions(Not these — that's tool use/agents)
Which lever solves which problem
Press Enter or Space to flip the card. Use the left and right arrow keys to move between cards.Term shown.
1 / 4

Why people get it wrong​

Fine-tuning sounds like "teaching the model," so it feels like the real fix. But it's the slowest, costliest, least flexible option, it doesn't add fresh knowledge well (RAG does that), and it's easy to do badly. Exhaust prompting and RAG first — you usually won't need step 3.

Watch out
  • Fine-tuning a frontier model on your data does NOT reliably add new facts — it shifts style and behavior. Facts still hallucinate.
  • A fine-tuned model is pinned to that base version. When the vendor ships a better base, you re-train or you're stuck.
  • Bad training data amplifies bad behavior — a mediocre dataset makes the model worse, not better.
  • Most 'we need fine-tuning' requests are really 'our prompt is 40 lines of ad-hoc rules' — refactor the prompt first.

:::tip They combine A strong system is often a good prompt + RAG for knowledge, with fine-tuning reserved for a narrow behavioral need. They're not mutually exclusive. :::

A concrete example: 'the model can't answer questions about our product'

# Wrong first move: fine-tune on the product manual.
# Right first move: retrieve the relevant chapter and inject it.

# 1. PROMPTING (baseline)
System: "You are a support agent for Acme Widgets. Be concise, cite doc sections."
User:   "Does the Widget Pro support Bluetooth 5.3?"
# Fails — model has no idea what a Widget Pro is.

# 2. + RAG (usually enough)
System: "You are a support agent for Acme Widgets. Be concise, cite doc sections."
Context: [top-3 chunks from product manual retrieved by embedding search]
User:    "Does the Widget Pro support Bluetooth 5.3?"
# Works — model reads the retrieved specs and answers with a citation.

# 3. + Fine-tuning (only if house voice / format is still drifting after 1+2)
# Train on 500-2000 (prompt, ideal answer) pairs to lock in the style — NOT to teach facts.

Check yourself

0/4
  1. The model gives vague, off-format answers to your task. What do you try first?
  2. Your model doesn't know about your internal wiki. Which lever?
  3. What is fine-tuning ACTUALLY good at?
  4. Which combination is the shape of a mature production system?
Key takeaways
  • Three levers when the model isn't doing what you want: prompting, RAG, fine-tuning — in that order.
  • Prompting fixes ~9 out of 10 'model is bad at X' complaints for zero cost and instant iteration.
  • RAG is the right fix when the gap is missing or fresh knowledge — fine-tuning is NOT reliable for adding facts.
  • Fine-tuning is a last resort for consistent style/format/behavior at scale, and needs hundreds to thousands of high-quality examples plus the call volume to justify it.
  • In a real production system the three compose: strong prompt + RAG + narrow fine-tune.

Next​