Skip to main content

Sampling Controls: Temperature & Friends

Intermediate
What you'll learn
  • What temperature, top-p, top-k and stop sequences actually do to the next-token pick
  • A one-glance decision table for when to run cold vs hot
  • Why setting temperature=0 does NOT guarantee identical outputs
  • How to think about sampling on newer Claude models that hide the knobs

When a model generates text, it picks the next token from a probability distribution. Sampling controls tune how it picks — how focused vs. how varied the output is.

The main dials​

Guided walkthrough1 of 4
  1. Scales the whole distribution. Low (≈0) = focused, near-deterministic, repeatable; the model takes the most likely path. High = more varied and creative, but more prone to wandering or error.
Pro tip
  • Adjust temperature OR top-p, not both — they interact and the combined behavior is hard to reason about.
  • Leave top-k alone by default. It's a heavy hammer that mostly gets in the way.
  • Stop sequences are a great safety belt for structured output — no more trailing prose after your JSON.

When to run cold vs hot​

Run cold (low temp)Run hot (higher temp)
Extraction, classification, codeBrainstorming, names, creative copy
Anything you want reproducibleExploring many options
Factual / structured outputTone variety, ideation
Grading / LLM-as-judgeIdeation for A/B copy

A good default for most work is moderate-to-low (roughly 0–0.4). Crank it up only when you want surprise.

A worked example​

Same prompt, three temperatures. Illustrative pattern, not a benchmark:

Prompt

Give me one clever name for a productivity app that helps writers.
  • temp 0 → a safe, repeatable pick like WordFlow. Ask again, get WordFlow again.
  • temp 0.5 → still coherent, more variety across runs: Draftly, Inkline, Composera.
  • temp 1.0 → creative but noisier: Quillspark, Verbatimo, occasionally a duff word that doesn't scan.

The lesson: crank temperature only where variety is the point of the task.

Newer Claude models hide these​

Several recent Claude models adapt their own decoding and de-emphasize (or omit) temperature. If a knob isn't available, that's by design — shape behavior through the prompt and (where offered) the effort/thinking setting instead. Prompt clarity and role framing do more work than sampling knobs on modern models.

Determinism caveat​

Even at temperature 0, outputs are not guaranteed bit-identical across runs, model versions, or providers. Reasons:

  • Batching and floating-point non-determinism on the server side.
  • Silent model updates behind an alias.
  • Slight input differences (a stray whitespace, a different tokenizer).

Don't rely on exact reproducibility. Rely on evals to catch drift when it matters.

Common mistakes​

Pro tip
  • Cranking temperature to fix a bad output — the fix is almost always in the prompt, not the sampler.
  • Mixing top-p and temperature — pick one knob and keep the other at its default so you can reason about changes.
  • Assuming temp=0 means deterministic — it does not. Use evals, not equality checks, to detect regressions.
  • Using stop sequences on prose — great for JSON delimiters, awkward for freeform text where the model might legitimately produce the string.

Check yourself​

Check yourself

0/4
  1. You need to extract fields from an invoice reliably. Which temperature setting?
  2. Which pair of knobs should you generally NOT tune together?
  3. You set temperature=0 and get slightly different outputs on two runs. What's the most likely cause?
  4. On a newer Claude model where the temperature knob isn't exposed, how do you shape behavior?
Key takeaways
  • Temperature is the main knob; top-p is an alternative; top-k and stop sequences are situational.
  • Cold for extraction, code, and grading; hot for brainstorming and creative copy.
  • Adjust one randomness knob at a time — temperature OR top-p.
  • Temp 0 ≠ deterministic. Rely on evals, not equality, to catch behavior drift.
  • On newer Claude models, prompt + effort/thinking replace the sampling dials.

Next​