Skip to main content

Sampling Controls: Temperature & Friends

Intermediate
What you'll learn
  • What temperature, top-p, top-k and stop sequences actually do to the next-token pick
  • A one-glance decision table for when to run cold vs hot
  • Why setting temperature=0 does NOT guarantee identical outputs
  • How to think about sampling on newer Claude models that hide the knobs

When a model generates text, it picks the next token from a probability distribution. Sampling controls tune how it picks — how focused vs. how varied the output is.

The main dials​

Guided walkthrough1 of 4
  1. Scales the whole distribution. Low (≈0) = focused, near-deterministic, repeatable; the model takes the most likely path. High = more varied and creative, but more prone to wandering or error.
Pro tip
  • Adjust temperature OR top-p, not both — they interact and the combined behavior is hard to reason about.
  • Leave top-k alone by default. It's a heavy hammer that mostly gets in the way.
  • Stop sequences are a great safety belt for structured output — no more trailing prose after your JSON.

When to run cold vs hot​

Run cold (low temp)Run hot (higher temp)
Extraction, classification, codeBrainstorming, names, creative copy
Anything you want reproducibleExploring many options
Factual / structured outputTone variety, ideation
Grading / LLM-as-judgeIdeation for A/B copy

A good default for most work is moderate-to-low (roughly 0–0.4). Crank it up only when you want surprise.

A worked example​

Same prompt, three temperatures. Illustrative pattern, not a benchmark:

Prompt

Give me one clever name for a productivity app that helps writers.
  • temp 0 → a safe, repeatable pick like WordFlow. Ask again, get WordFlow again.
  • temp 0.5 → still coherent, more variety across runs: Draftly, Inkline, Composera.
  • temp 1.0 → creative but noisier: Quillspark, Verbatimo, occasionally a duff word that doesn't scan.

The lesson: crank temperature only where variety is the point of the task.

Newer Claude models hide these​

Several recent Claude models adapt their own decoding and de-emphasize (or omit) temperature. If a knob isn't available, that's by design — shape behavior through the prompt and (where offered) the effort/thinking setting instead. Prompt clarity and role framing do more work than sampling knobs on modern models.

Determinism caveat​

Even at temperature 0, outputs are not guaranteed bit-identical across runs, model versions, or providers. Reasons:

  • Batching and floating-point non-determinism on the server side.
  • Silent model updates behind an alias.
  • Slight input differences (a stray whitespace, a different tokenizer).

Don't rely on exact reproducibility. Rely on evals to catch drift when it matters.

That first reason is worth getting precise, because the usual shorthand ("GPUs and floating point are random") is wrong and makes the problem sound unfixable. The same kernel on the same data returns bitwise identical results every time — what changes is the batch shape a shared server hands it, which changes the reduction order, which changes the last bits. Your numerics move because other people's traffic moved. See batch invariance for the mechanism, the engines that now let you switch it off, and the measurement that found 80 unique completions in 1,000 temperature-0 runs of one prompt.

Common mistakes​

Pro tip
  • Cranking temperature to fix a bad output — the fix is almost always in the prompt, not the sampler.
  • Mixing top-p and temperature — pick one knob and keep the other at its default so you can reason about changes.
  • Assuming temp=0 means deterministic — it does not. Use evals, not equality checks, to detect regressions.
  • Using stop sequences on prose — great for JSON delimiters, awkward for freeform text where the model might legitimately produce the string.

Check yourself​

Check yourself

0/4
  1. You need to extract fields from an invoice reliably. Which temperature setting?
  2. Which pair of knobs should you generally NOT tune together?
  3. You set temperature=0 and get slightly different outputs on two runs. What's the most likely cause?
  4. On a newer Claude model where the temperature knob isn't exposed, how do you shape behavior?
Key takeaways
  • Temperature is the main knob; top-p is an alternative; top-k and stop sequences are situational.
  • Cold for extraction, code, and grading; hot for brainstorming and creative copy.
  • Adjust one randomness knob at a time — temperature OR top-p.
  • Temp 0 ≠ deterministic. Rely on evals, not equality, to catch behavior drift.
  • On newer Claude models, prompt + effort/thinking replace the sampling dials.

Next​