Skip to main content

Diffusion LLMs: The Models That Don't Write Left to Right

Advanced

Every model on AILmanac so far generates text the same way: one token, then the next, left to right, forever. A different family of models does not. It starts with a blank canvas of masked positions, predicts all of them at once, keeps the ones it is confident about, and repeats. Position 40 can be decided before position 7.

The numbers are the reason anyone cares. Inception's Mercury 2 — a reasoning model — reportedly runs at 1,009 tokens/sec on Blackwell GPUs with 1.7 s end-to-end latency, roughly 8–14× faster than comparable reasoning-enabled competitors (The Decoder). Google's experimental Gemini Diffusion samples at 1,479 tokens/sec excluding overhead, against Gemini 2.5 Flash's 272.4 (Techmeme roundup).

And yet the README of LLaDA — the 8B open-weight model that proved the approach at scale — tells you its quality peaks when the number of sampling steps equals the response length. One forward pass per token. Exactly like an autoregressive model, but without a KV cache, so slower.

Both facts are true. This lesson is the bridge between them: what the algorithm actually does, which knob buys the speed and what it costs, the two structural limits that made early diffusion LLMs unusable and the fix that shipped, and the three gotchas (streaming, length billing, and a jailbreak class that does not exist on autoregressive models) you need before you put one in production.

What you'll learn
  • Describe masked-diffusion generation in terms of a canvas, a denoising pass, and a commit decision — and say what bidirectional attention buys and costs
  • Use the steps knob correctly: compute tokens-per-forward-pass, and explain the conditional-independence failure that makes aggressive parallelism wrong rather than merely worse
  • Explain why block diffusion is on every dLLM you can actually call today — it is what restored the KV cache, variable-length output, and token streaming
  • Pick the jobs where a dLLM genuinely wins: infilling and any-order edits, latency-bound voice, and bounded decisions — not 'everything, but faster'
  • Treat an open dLLM endpoint as unaligned at the boundary, because masked-infill prompts bypass alignment in a way autoregressive refusals cannot

The mechanism, in one diagram you have to imagine​

An autoregressive (AR) model's loop is: read the whole prefix, emit a probability distribution over one position (the next one), sample, append, repeat. The KV cache is what makes this cheap — the keys and values for every earlier token were computed once and never change, because attention is causal and nothing to the right exists yet.

A masked diffusion LLM inverts that. Attention is bidirectional: every position attends to every other position, in both directions. So in a single forward pass the model produces a distribution for every masked position simultaneously. Generation becomes an iterative refinement of a whole answer, not a walk along it.

Guided walkthrough1 of 5
  1. Decide a generation length L. Fill all L positions with a special mask token. On a pure masked dLLM (LLaDA, Dream), L is fixed before the first forward pass — the model does not grow the sequence as it goes.

That last line is the one worth memorising. An AR model is pinned at one token per forward pass. A diffusion LLM lets you choose. DiffusionGemma, Google's open-weight entry, reports roughly 20 tokens per forward pass while refining blocks of 256 in parallel, which is how it reaches ~1,500 output tokens/sec on a single H100 (DiffusionGemma Technical Report).

The steps knob, and why cranking it is wrong rather than just lossy​

Dream 7B exposes the knob explicitly. Its diffusion_generate() documents steps as "diffusion timesteps" where max_new_tokens / steps tokens are produced per step, and notes plainly that fewer steps are faster but coarser (DreamLM/Dream, Apache-2.0).

So why not always use few steps? Because the error it introduces is not noise — it is a specific, nameable bug.

A single forward pass gives you, for each masked slot, a marginal distribution: the probability of a token at that position, averaged over everything still unknown. If you commit two slots from the same pass, you are sampling from the product of their marginals, not from their joint distribution. Any dependency between them is silently discarded. The Fast-dLLM authors state the root cause directly: generation quality degrades under parallel decoding because masked diffusion models sample tokens independently from marginal probabilities when the true joint distribution has dependencies (Fast-dLLM, arXiv:2505.22618).

The intuition: ask for a city and two adjacent slots might score "New" and "Delhi" as the most likely token for their own position, because "New York" and "New Delhi" are both plausible. Commit both in one pass and you get neither. The failure mode of over-parallel decoding is not vaguer prose, it is confidently assembled nonsense at the seams.

The fix that made the field practical is confidence-aware parallel decoding: compute a confidence per masked slot (typically the max softmax probability), commit only the slots above a threshold, and if none clear it, commit the single highest so the loop still makes progress. Tokens that are nearly independent get decoded together; tokens that depend on each other wait. Fast-dLLM reports up to 27.6× throughput improvement on LLaDA and Dream across benchmarks with minimal accuracy loss — training-free, just a decoding change.

That is the shape of this entire research area: the parallelism is free, the independence is not, and the work is in detecting which tokens can safely go together.

What you'll learn
  • A confidence threshold is a quality dial you can actually reason about: higher threshold means fewer tokens committed per pass, more passes, slower, closer to AR quality. Fast-dLLM v2 reports a 2.6× speedup on GSM8K at threshold 0.9 with minimal accuracy loss — that is the kind of setting to start from, not 'commit everything'.

The two structural limits — and block diffusion, which is why you can call one today​

LLaDA's own repository lists its efficiency constraints without euphemism: fixed context length for sampling, no KV-cache support, and degraded performance when sampling steps drop below the response length. The authors frame it as deliberate — they wanted to find the ceiling of the approach before optimising it (ML-GSAI/LLaDA, MIT, ~4k stars, Base and Instruct weights plus LLaDA 1.5, LLaDA-MoE-7B-A1B and iLLaDA variants).

Stack those up and a pure masked dLLM is nearly unusable as a product:

  1. No KV cache. Bidirectional attention means every pass can change any token's context, so cached keys and values are invalid. You recompute the whole sequence every step.
  2. Fixed output length. You must pick L before generating. Too small truncates; too large wastes compute on positions you will throw away.
  3. Latency worse than AR. Points 1 and 2 together can make a "parallel" model slower end-to-end than the sequential one it replaces.

Block diffusion resolves all three by refusing to be pure. Generate in blocks: condition on previous blocks autoregressively, run diffusion inside the current block. That gives flexible sequence length and KV caching between blocks — the BD3-LM formulation, an ICLR 2025 oral (arXiv:2503.09573, kuleshov-group/bd3lms).

Fast-dLLM v2 (arXiv:2509.26328) shows what a tuned version looks like, and the details are worth knowing because they explain the vendor behaviour you will observe:

  • Hybrid attention mask. Block-diagonal (bidirectional) within a block, block-causal (left-to-right) between blocks. Bidirectional context where it helps, AR semantics where the cache lives.
  • Two cache levels. A block-level cache holding finished blocks as clean prefix context, plus a sub-block cache (DualCache) for the recomputation that intra-block decoding still needs.
  • Complementary masking in training. Each sample is duplicated into two views with masks m and 1 − m, so every token is supervised both as a mask target and as context.
  • Reported results. Up to 2.5× faster than standard AR decoding; 1.8× at batch size 64 on H100s; a 7B average benchmark score of 60.3 against Dream's 57.6, including HumanEval 63.4 vs 57.9 and GSM8K 62.0 vs 39.2.

And the training cost is the detail that reframes the field: Fast-dLLM v2 reports fine-tuning from an AR model on roughly 1B tokens — about 500× less than Dream's 580B — a 7B trained in ~12 hours on 64 A100s.

Diffusion is a decoding mode you can retrofit, not a separate species​

DiffusionGemma makes that point unmissable. It is not trained from scratch: it fine-tunes Gemma 4, a mixture-of-experts model with 3.8B activated and 25.2B total parameters, into a diffusion model using fewer than 10% of the starting AR model's total training token budget — supervised denoising fine-tuning, then reinforcement learning and sampler distillation (arXiv:2608.00146, CC-BY-4.0 weights).

What survives the conversion is the surprising part. The report states it retains the base model's thinking mode, multimodal inputs and long contexts — and keeps the ability to generate autoregressively with only minor degradation. One set of weights, two decoding modes.

If that result holds up, "diffusion LLM" stops being a distinct product category and becomes a serving option: the same family, with a parallelism dial on the inference side. That is a far more consequential claim than any single tokens-per-second record, and it is the reason to learn the mechanism now rather than when a provider flips a flag on a model you already use.

What a diffusion LLM is actually better at​

Speed is the headline and the weakest part of the case, because speed is contested — specialised AR hardware is also delivering hundreds of tokens per second (see Token Speed). The durable advantages are the ones that follow from the shape of the algorithm.

Any-order generation and native infilling. Because you choose which slot to commit, you can leave a hole in the middle of a sequence and have the model fill it with full context on both sides. Dream exposes this as the alg parameter — origin (random order), maskgit_plus (top-1 confidence), topk_margin (top1 − top2 margin), entropy — plus alg_temp to add randomness to those strategies. For code edits, template completion, or filling fields in a fixed JSON skeleton, that is a structurally better fit than asking an AR model to re-emit a whole file to change line 40.

Latency-bound interaction. Inception's Mercury Voice advertises time-to-first-token under 170 ms on a 128K context, priced at $0.20/$1.50 per million input/output tokens at the time of writing. Voice is the clearest case where latency is the product; our full-duplex voice AI page covers why that threshold matters.

End-to-end reasoning turnaround. Mercury 2's reported 1.7 s for a complete reasoning answer is a different claim from throughput — it is the number an agent loop actually feels, turn after turn.

Bounded answers instead of prose. Inception's Mercury Decide returns a choice, a score, or a yes/no with a calibrated probability read off the model rather than written out as text — reported at up to 14 decisions per second, which makes "run it on every case and escalate the unsure ones" affordable. The category and its caveats are covered in depth on System One decision models; the diffusion backbone is an implementation detail there, not the point.

Price pressure. Mercury 2.5 currently lists a 260K context at $0.04/$0.15 per million input/output tokens (promotional, off $0.20/$0.75) with cached input at $0.004 per million. Treat promo pricing as temporary and model your bill at list.

Three gotchas before you ship one​

Watch out
  • Tokens/sec is measured at the vendor's chosen parallelism. Your quality bar may require more steps, which costs you the speedup. Benchmark on your own prompts at your own accuracy threshold.
  • Output is not final left-to-right. Only block boundaries are monotonic; inside a block, slots can still be masked or revised. Any UI or parser that assumes 'token N will never change' is wrong.
  • A pure masked dLLM bills and computes a length you chose in advance. Block diffusion reduces that to per-block granularity — it does not remove it.

The third one deserves more than a bullet, because it is a security property and it has no autoregressive analogue.

An AR model's alignment has a structural advantage: the refusal happens first. If the model decides to decline, every subsequent token is conditioned on "I can't help with that." A diffusion LLM has no such moment. There is no left-to-right point at which a refusal takes over the rest of the generation — masked slots get filled to be consistent with whatever is already on the canvas.

DIJA ("The Devil behind the mask", ICLR 2026, arXiv:2507.11097, ZichenWen1/DIJA) turns that into an attack. The adversarial prompt puts the harmful intent in unmasked tokens and leaves masked spans for the model to complete in a contextually consistent way. The reported results: up to 100% keyword-based attack success rate on Dream-Instruct, surpassing the strongest prior baseline by up to 78.5% in evaluator-based ASR on JailbreakBench and by 37.7 points in StrongREJECT score — while requiring no rewriting or obfuscation of the harmful content. The authors' conclusion is that existing alignment mechanisms do not protect dLLMs against context-aware masked-input prompts, because the model cannot filter an unsafe generation mid-inference the way a sequential one can.

A follow-up (arXiv:2602.00388) argues the picture is two-sided — diffusion confers a safety blessing in some settings that then breaks under context — which is a reason to measure rather than to assume either way.

Practical takeaway: if you expose an open-weight dLLM, do not treat its instruction tuning as a guardrail. Filter and classify at the boundary, on both the request and the response, exactly as you would for a base model. Our securing local and hybrid agents page covers the boundary pattern.

Try one in ten minutes​

Guided walkthrough1 of 4
  1. Inception serves Mercury through an OpenAI-compatible API, and the models are listed on OpenRouter (for example openrouter.ai/inception/mercury-2). Point an existing OpenAI-SDK script at it and change the model string — nothing about the request shape is diffusion-specific.

Call a hosted diffusion LLM with the OpenAI SDK (nothing changes but the model string)

# Inception serves an OpenAI-compatible API; OpenRouter also lists the models.
# Check the provider's model list for the current slug before you hard-code one.

curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
  "model": "inception/mercury-2",
  "messages": [
    {"role": "user", "content": "Rewrite this function to stream its output, and explain the two failure modes you introduced."}
  ],
  "stream": true
}'

# What to watch for, beyond the content:
#  - time to first token vs time to last token (the diffusion win is usually in the gap)
#  - whether the stream arrives in chunks that look like BLOCKS rather than words

The steps sweep on Dream 7B — find your own quality cliff

import time, torch
from transformers import AutoModel, AutoTokenizer

NAME = "Dream-org/Dream-v0-Instruct-7B"
tok = AutoTokenizer.from_pretrained(NAME, trust_remote_code=True)
model = AutoModel.from_pretrained(
  NAME, torch_dtype=torch.bfloat16, trust_remote_code=True
).to("cuda").eval()

PROMPT = "List the five largest cities in Germany with their populations."
MAX_NEW = 256

# steps == MAX_NEW is the authors' best-quality setting: one token per
# forward pass. Each halving doubles tokens-per-pass and doubles the
# number of independence assumptions you are making.
for steps in [256, 128, 64, 32, 16]:
  inputs = tok.apply_chat_template(
      [{"role": "user", "content": PROMPT}],
      return_tensors="pt", return_dict=True, add_generation_prompt=True,
  ).to("cuda")

  t0 = time.perf_counter()
  out = model.diffusion_generate(
      inputs.input_ids,
      attention_mask=inputs.attention_mask,
      max_new_tokens=MAX_NEW,
      steps=steps,
      temperature=0.0,
      alg="entropy",          # commit the lowest-entropy slots first
      alg_temp=0.0,
  )
  dt = time.perf_counter() - t0
  text = tok.decode(out[0][len(inputs.input_ids[0]):], skip_special_tokens=True)

  print(f"--- steps={steps}  tokens/pass={MAX_NEW/steps:.0f}  {dt:.2f}s ---")
  print(text.strip()[:400])

# Read the OUTPUTS, not just the timings. The numbers and the city names are
# the multi-token units that break first when too many slots are committed
# from one pass. That corruption is the conditional-independence bug, visible.

How to decide whether this matters to you​

Key takeaways
  • A diffusion LLM's speed comes from one ratio — generation length divided by denoising steps, i.e. tokens committed per forward pass. An AR model is fixed at 1. Everything else is a consequence of that dial.
  • Over-parallel decoding is wrong, not just coarse: committing correlated slots from one pass samples the product of marginals instead of the joint. Confidence-threshold decoding is the fix, and Fast-dLLM reports up to 27.6x throughput with minimal accuracy loss.
  • Pure masked diffusion has no KV cache and a fixed output length. Block diffusion (AR between blocks, diffusion inside) is what restored caching, variable length and streaming — it is why every callable dLLM today is a block model.
  • DiffusionGemma converted an autoregressive Gemma 4 MoE into a diffusion model for under 10% of its original training token budget while keeping thinking mode, multimodality and AR generation. Diffusion is looking more like a decoding mode than a separate species of model.
  • Pick it for infilling and any-order edits, sub-200 ms voice turns, fast bounded decisions and end-to-end turn latency — and assume its alignment does not hold, because masked-infill prompts reach up to 100% ASR on an aligned open dLLM.
Press Enter or Space to flip the card. Use the left and right arrow keys to move between cards.Term shown.
1 / 11

Check yourself

0/5
  1. A masked diffusion LLM is asked for a 256-token answer in 32 denoising steps. What does that imply?
  2. Why can't a pure masked diffusion LLM use a conventional KV cache?
  3. What does block diffusion (BD3-LM) change relative to pure masked diffusion?
  4. DiffusionGemma's most consequential reported result is which of these?
  5. You are exposing an open-weight, instruction-tuned dLLM to untrusted users. What is the dLLM-specific risk?

New to the underlying machinery? What is an LLM and Tokens and context cover the autoregressive baseline this page departs from, and Token Speed covers the hardware and software multipliers that attack the same latency problem from the other side.

Sources & further reading​

Next​