DPO

Direct Preference Optimization for image models

↓ scroll to learn ↓

The Big Idea

We have an image generator and lots of pairs of images for the same caption: one we like 👍 (say, a real photo) and one we don't 👎 (say, an image with a plastic, “AI-looking” finish). DPO nudges the model to get better at the 👍 image and worse at the 👎 image, always measured against a frozen copy of where it started.

Caption c

Image A
🧑‍⚖️which is better?
Image B

The plan. We follow the math exactly as a real training loop computes it, one small idea per section:

  1. 0Notation
  2. 1The per-image diffusion loss
  3. 2Four numbers per pair
  4. 3Reward and margin
  5. 4The loss
  6. 5Where it comes from
  7. 6What the gradient says about β
  8. 7One full training step

0 Notation: Meet the Cast

Only six symbols. Every equation on this page is built from these.

\(c\)
📝

Caption

The text prompt. Both images in a pair share the same caption.

\(x^w\)
👍

Winner

The preferred image (e.g. a real photo), stored as VAE latents.

\(x^l\)
👎

Loser

The rejected image.

\(\theta\)
🧠

Student

The model being trained. It starts as a copy of the base model.

\(\text{ref}\)
🧊

Reference

A frozen copy of that same base model (often called the “teacher”). It is never updated.

\(\beta\)
🎚️

Beta

One number that sets how hard, and for how long, DPO pushes. Section 6 is all about it.

1 The Per-Image Diffusion Loss

Before any DPO, we need one number per image: how well can the model clean up a noisy version of it? This is the ordinary training loss of a rectified-flow (flow-matching) image model.

Noise the image, then ask for the velocity
\[ x_t = (1-t)\,x_0 + t\,\varepsilon, \qquad v = x_0 - \varepsilon \] \[ \ell_\theta(x_0, c;\,t,\varepsilon) = \operatorname{mean}_{\text{tokens, channels}}\;\big\|\,v_\theta(x_t, t, c) - v\,\big\|^2 \]

Pick a time \(t\in(0,1)\) and Gaussian noise \(\varepsilon\sim\mathcal N(0,I)\). Mix them in a straight line, ask the model for the velocity \(v\), and score it with the mean squared error. Lower \(\ell\) means the model “understands” this image better.

clean image \(x_0\)
noise \(\varepsilon\)
noisy input \(x_t\) (what the model sees)
true velocity \(v = x_0 - \varepsilon\)
model's guess \(v_\theta\)
squared error \((v_\theta - v)^2\)
per-image loss ℓ (mean of the error map)0

Slide t to move between the clean image (t = 0) and pure noise (t = 1). The model error slider plays the role of a better or worse model: a perfect model (0) gives a black error map and ℓ = 0. In real training, images are VAE latents rather than pixels, but the arithmetic is identical. This is the same straight-line idea as the Flow Matching explainer, with time running the other way.

🎲 Which t gets picked?

\(t\) is not uniform. It's drawn from a logit-normal(0, 1) distribution (\(t = \sigma(n)\) with \(n\sim\mathcal N(0,1)\)), which favors the middle, where denoising is hardest and most informative. It is then shifted toward noisier \(t\) by a factor that depends on resolution, because bigger images need more noise to hide their content.

Each falling dot is one sampled \(t\). The curve is the exact density. A common form of the shift is \(t' = \dfrac{s\,t}{1 + (s-1)\,t}\) (used in Stable Diffusion 3). At \(s = 1\) nothing changes; larger \(s\) slides the mass toward \(t = 1\), i.e. more noise.

2 Shared Noise: Four Numbers per Pair

For each pair, draw one \(t\) and one \(\varepsilon\), and reuse them for both the winner and the loser. Then run the frozen reference on exactly the same inputs, with no gradients. That gives four losses per pair. Step through it:

clean \(x_0\)
noise \(\varepsilon\)
noisy \(x_t\)
👍 \(x^w\)
same ε
👎 \(x^l\)
same ε

👍 winner \(x^w\)👎 loser \(x^l\)
🧠 student \(\theta\) (trainable)\(L^\theta_w = 0.30\)\(L^\theta_l = 0.50\)
🧊 reference (frozen, no gradients)\(L^{\text{ref}}_w = 0.40\)\(L^{\text{ref}}_l = 0.40\)
\(L^\theta_w = \ell_\theta(x^w),\;\; L^{\text{ref}}_w = \ell_{\text{ref}}(x^w),\;\; L^\theta_l = \ell_\theta(x^l),\;\; L^{\text{ref}}_l = \ell_{\text{ref}}(x^l)\), all at the same \((t, \varepsilon)\)

🎲 Why share the noise?

A denoising loss depends heavily on luck: which \(t\) and which \(\varepsilon\) were drawn. If the winner and the loser got different noise, comparing their losses would mostly compare dice rolls. With the same \((t,\varepsilon)\) for both images and for both models, every difference we compute next reflects the models, not the noise. (Training logs often report winner_loss and loser_loss: these are the batch averages of \(L^\theta_w\) and \(L^\theta_l\).)

3 Reward and Margin

Compare the student with the reference on each image, then compare the two images:

\[ r_w = L^\theta_w - L^{\text{ref}}_w, \qquad r_l = L^\theta_l - L^{\text{ref}}_l, \qquad \Delta = r_w - r_l \]

\(r < 0\) means the student fits that image better than the reference does. The goal: fit the winner better (\(r_w\downarrow\)) and the loser worse (\(r_l\uparrow\)), so \(\Delta\) becomes strongly negative.

\(r_w\) (winner vs reference)−0.10
\(r_l\) (loser vs reference)+0.10
margin \(\Delta = r_w - r_l\)−0.20

Why measure against a reference at all? Some images are simply easier to denoise than others. If we compared raw losses, the model could look “good” just because the 👍 image happens to be easy. Subtracting the reference's loss on the same image, with the same noise, cancels out that built-in difficulty. Only the change the student has made since training began is left.

4 The Loss

One line turns the margin into something to minimize:

The DPO loss for one pair (averaged over the batch)
\[ \mathcal{L} = -\log\sigma(-\beta\,\Delta) \]

\(\sigma(z) = \frac{1}{1+e^{-z}}\) is the sigmoid, which squashes any number into (0, 1).

🧮 A friendlier way to write it: softplus

Since \(\sigma(-z) = \frac{1}{1+e^{z}}\), taking \(-\log\) gives

\[ -\log\sigma(-z) = \log\big(1 + e^{z}\big) = \operatorname{softplus}(z) \quad\Longrightarrow\quad \mathcal{L} = \operatorname{softplus}(\beta\,\Delta) = \log\big(1 + e^{\beta\Delta}\big) \]

Softplus is a smooth version of \(\max(0, z)\): almost 0 when \(z\) is very negative, almost \(z\) when it's very positive. So the loss is near 0 once \(\beta\Delta\) is well below 0, and it grows when the margin goes the wrong way.

loss \(\operatorname{softplus}(\beta\Delta)\)0.693
slope \(d\mathcal{L}/d\Delta = \beta\,\sigma(\beta\Delta)\)1.000

🧪 Every run starts at 0.693

At step 0 the student is the reference, so every \(r = 0\), \(\Delta = 0\), and \(\mathcal L = \log(1 + e^0) = \ln 2 \approx 0.693\). If your first logged DPO loss isn't 0.693, your reference is not the checkpoint you started from.

🧊 β = 0 means no learning

With \(\beta = 0\) the loss is the constant \(\ln 2\) no matter what \(\Delta\) is: a flat curve, zero gradient. Drag β to 0 in the plot above to see it flatten.

5 Where This Comes From (Diffusion-DPO)

DPO was first written for language models, which can compute exact log-probabilities. Image diffusion models can't, but their training loss stands in for one. Step through the substitution:

① DPO for language models
\[ \mathcal L = -\log\sigma\!\Big(\beta_0\Big[\log\frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)} - \log\frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)}\Big]\Big) \]

Compare how much more likely the student makes the winner vs. the loser, relative to the reference.

② Problem: no exact likelihood for images
\[ \log p_\theta(x\mid c) \;\approx\; -k\;\mathbb{E}_{t,\varepsilon}\big[\ell_\theta(x)\big] + \text{const} \]

The evidence lower bound (ELBO) says: log-likelihood ≈ minus a constant times the denoising loss. Lower loss means more likely.

③ So a log-ratio becomes a loss difference
\[ \log\frac{p_\theta(x)}{p_{\text{ref}}(x)} \;\approx\; -k\,\big(L^\theta - L^{\text{ref}}\big) = -k\,r \]

The “const” is the same for both models, so it cancels. What's left is exactly our reward \(r\) from section 3.

④ Substitute into ①
\[ \beta_0\big[(-k\,r_w) - (-k\,r_l)\big] = -\beta_0 k\,(r_w - r_l) = -\beta\,\Delta, \qquad \beta = k\,\beta_0 \]
⑤ Result: the loss from section 4
\[ \mathcal L = -\log\sigma(-\beta\,\Delta) \]

In code it's one line: -logsigmoid(-beta * reward_diff). This is Diffusion-DPO (Wallace et al., 2023).

Optional: why does language-model DPO (step ①) look like that? →
Step A

A model of “A beats B” (Bradley–Terry)

Give every output a hidden score \(r(y)\) and assume people prefer \(y_w\) over \(y_l\) with probability

\[P(y_w \succ y_l) = \sigma\big(r(y_w) - r(y_l)\big)\]
Step B

The goal: high score, but stay near the reference

\[\max_{\pi}\; \mathbb{E}_{y\sim\pi}\big[r(y)\big] - \beta_0\,\mathrm{KL}\big(\pi\,\|\,\pi_{\text{ref}}\big)\]
Step C

It has an exact solution

Rewrite the objective as \(-\beta_0\,\mathrm{KL}(\pi\,\|\,\pi^*) + \beta_0\log Z\). A KL divergence is smallest (zero) when the two distributions match, so the best model is

\[\pi^*(y) = \frac{1}{Z}\,\pi_{\text{ref}}(y)\,e^{\,r(y)/\beta_0},\qquad Z = \sum_y \pi_{\text{ref}}(y)\,e^{\,r(y)/\beta_0}\]
Step D

Flip it: the reward hides inside the model

\[r(y) = \beta_0\log\frac{\pi^*(y)}{\pi_{\text{ref}}(y)} + \beta_0\log Z\]
Step E

Plug into Bradley–Terry: \(Z\) cancels

\[P(y_w\succ y_l) = \sigma\Big(\beta_0\log\frac{\pi^*(y_w)}{\pi_{\text{ref}}(y_w)} - \beta_0\log\frac{\pi^*(y_l)}{\pi_{\text{ref}}(y_l)}\Big)\]

Replace the unknown \(\pi^*\) by the student \(\pi_\theta\) and minimize the negative log-likelihood of the observed preferences. That's exactly step ①. No separate reward model is needed.

6 The Gradient Explains Everything About β

The reference losses are constants (no gradients flow through the frozen model), so \(\nabla\Delta = \nabla L^\theta_w - \nabla L^\theta_l\). With the chain rule and \(\frac{d}{dz}\operatorname{softplus}(z) = \sigma(z)\):

\[ \nabla_\theta \mathcal L = w\,\big(\nabla_\theta L^\theta_w - \nabla_\theta L^\theta_l\big), \qquad w = \beta\,\sigma(\beta\Delta) \]

Gradient descent moves against this: the winner's loss is pulled down and the loser's loss is pushed up, both with strength \(w\).

🚀 The push starts at β/2

At step 0, \(\Delta = 0\) and \(\sigma(0) = \tfrac12\), so \(w = \beta/2\).

🔌 The push switches itself off

As \(\Delta\) goes negative, \(\sigma(\beta\Delta)\to 0\). Once \(\beta\Delta\approx -5\), \(\sigma \approx 0.007\) and the push is essentially off. That happens at \(|\Delta| \approx 5/\beta\).

👍 pull \(L^\theta_w\) down
👎 push \(L^\theta_l\) up
initial push β/20.5
switch-off point |Δ| ≈ 5/β5
current push w0.5
push left (% of start)100%

So what is β? It's the “stay near the reference” knob, the role the KL coefficient plays in standard DPO. Large β gives a strong but short push: the model separates winner from loser with a small change, and then the push stops on its own. Small β gives a gentle push that stays on until \(|\Delta|\) is huge (≈ 5/β). The loser's loss can be driven up enormously before anything stops it. Making realistic-looking images “bad” can damage the model's own outputs, so a too-small β can wreck a model.

What about gradient clipping? Clipping (e.g. to a max norm of 1.0) limits how big each step is, not which direction it goes. It slows a runaway push but doesn't stop it.

7 One Training Step, Animated

Everything above, in the order the computer actually does it. The highlighted step and the matching line of code move together.

The loop
1📦 Load a batch of (caption, 👍, 👎) pairs and encode both images with the VAE. E.g. 16 pairs = 32 images.
2🎲 For each pair, draw one t and one ε, and noise both images with them.
3🧠 Run the student on all noisy images: \(L^\theta_w,\ L^\theta_l\).
4🧊 Run the frozen reference on the same inputs, same t and ε, no gradients: \(L^{\text{ref}}_w,\ L^{\text{ref}}_l\).
5➖ Compute Δ, then \(\mathcal L = \operatorname{mean}\big[\operatorname{softplus}(\beta\Delta)\big]\).
6🔧 Backpropagate, clip the gradient norm (e.g. 1.0), and take an optimizer step (e.g. AdamW with a small learning rate). Only the student changes.
↺ repeat; the reference never changes
The code (PyTorch-style)
x_w, x_l = vae.encode(real), vae.encode(neg)t, eps = sample_t(), randn_like(x_w)  # shared!L_w = loss(student, x_w, c, t, eps)L_l = loss(student, x_l, c, t, eps)with torch.no_grad():    R_w = loss(reference, x_w, c, t, eps)    R_l = loss(reference, x_l, c, t, eps)delta = (L_w - R_w) - (L_l - R_l)dpo = F.softplus(beta * delta).mean()#   = -logsigmoid(-beta * delta)dpo.backward()clip_grad_norm_(student.parameters(), 1.0)optimizer.step(); optimizer.zero_grad()

🎛️ Same equation, three knobs

Every DPO variant you'll meet uses this exact equation. Only three things change:

🏁 The starting checkpoint

The student starts here, and the frozen reference is a copy of it.

👎 Where \(x^l\) comes from

Which images serve as losers: another model's outputs, an older checkpoint's, or the model's own.

🎚️ β

How hard and how long the push lasts (section 6).

Sanity Checks

🧪 First loss = 0.693?

If not, the reference and the starting checkpoint differ, so Δ isn't 0 at step 0. Use the same checkpoint for both.

🎲 Same t and ε everywhere?

Winner, loser, student and reference must all see identical \((t,\varepsilon)\) within a pair. Otherwise Δ is mostly noise.

📉 Watch winner and loser losses separately

The DPO loss can drop just because the loser's loss shoots up. Check that winner_loss isn't rising too, and always look at generated images.

🎯 Pairs teach whatever differs

If 👍 images are also brighter or bigger, the model learns “brighter” or “bigger”. Keep the main difference the one you care about.

Quick Recap: Flip the Cards

Click any card to reveal a plain-English definition.

\(x^w\) / \(x^l\)
Winner 👍 and loser 👎: two images for the same caption, as VAE latents.
Reference
A frozen copy of the starting model. Every change is measured against it.
\(\ell\) (diffusion loss)
Mean squared error between the predicted and true velocity on a noised image. Lower = model finds it more likely.
Shared noise
One \(t\) and one \(\varepsilon\) per pair, reused for both images and both models, so differences reflect the models.
Reward \(r\)
Student loss minus reference loss on one image. Negative = student is better on it.
Margin \(\Delta\)
\(\Delta = r_w - r_l\). Training drives it negative: better on 👍, worse on 👎.
\(\operatorname{softplus}(\beta\Delta)\)
The DPO loss, \(\log(1 + e^{\beta\Delta})\), the same as \(-\log\sigma(-\beta\Delta)\).
\(\ln 2 \approx 0.693\)
The loss at step 0, when student = reference. A great sanity check.
\(w = \beta\,\sigma(\beta\Delta)\)
Push strength. Starts at \(\beta/2\) and switches off once \(|\Delta| \approx 5/\beta\).

Want to go deeper?