DPO: Direct Preference Optimization

The Problem with Three Models

In mid-2023, an alignment engineer named Priya at a mid-sized AI company has been trying to run RLHF for six weeks. She has done everything right: 20,000 human preference pairs, a reward model that achieves 74% accuracy, PPO configured with all the right hyperparameters from the InstructGPT paper.

The training keeps crashing. Not from OOM - from PPO instability. The reward model score climbs for the first 200 steps, then the KL divergence explodes and the policy degenerates into incoherent outputs. She rolls back, adjusts the KL coefficient, runs again. It crashes differently. The loss is NaN. She debugs for three days - a single wrong tensor dtype in the reward computation was triggering numerical instability in the PPO update.

RLHF requires three separate models: the SFT model (frozen reference), the policy (being optimized), and the reward model. Each has its own forward pass in every training batch. The computational complexity, the numerical sensitivity, and the sheer number of moving parts make RLHF notoriously difficult to implement correctly.

Then a paper drops in May 2023. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." Rafael Rafailov and colleagues at Stanford and CRFSuite show that you can achieve the same alignment objective as RLHF - without training a reward model, without PPO, without three models in memory simultaneously. Just a language model, a reference model, and a simple binary cross-entropy loss.

Priya reads it over a weekend. Monday, she has a working implementation. The model quality matches her best RLHF attempt, achieved in one training run instead of a month of debugging.

Why This Exists: RLHF's Complexity Problem

RLHF is powerful but has three concrete engineering problems:

Problem 1: The reward model is a separate neural network that must be trained. This requires its own pipeline, its own preference data, its own evaluation, its own memory footprint. If the reward model is wrong (biased, miscalibrated, overtrained), all subsequent RLHF training is wrong too.

Problem 2: PPO is notoriously unstable for language models. Policy gradient methods like PPO were designed for game-playing agents with dense reward signals. Language generation has sparse rewards (one score per full response, not per token). This mismatch causes instability: large variance in gradients, sensitivity to hyperparameters, tendency to collapse or diverge.

Problem 3: Three models in memory simultaneously. The policy model (trainable), the reference model (frozen SFT), and the reward model - all loaded at the same time. For 7B+ models, this is prohibitively expensive.

DPO solves all three: no reward model, no RL, one trainable model plus one frozen reference.

Historical Context: The DPO Paper

"Direct Preference Optimization: Your Language Model is Secretly a Reward Model" was published by Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn in May 2023 (NeurIPS 2023).

The key insight that made DPO possible: the optimal policy under KL-constrained RLHF can be expressed directly as a function of the reward model and the reference model. This means you can implicitly train the reward model and the policy simultaneously, without ever explicitly instantiating the reward model.

The paper showed: on summarization (Reddit TL;DR) and dialogue (Anthropic HH-RLHF), DPO matched or outperformed PPO-based RLHF while requiring significantly less compute and being much more stable. The open-source community immediately adopted it - within months, most public instruction-tuned models used DPO for alignment.

The Core Derivation

This is the most important math in the lesson. If you understand this derivation, you understand why DPO works.

Step 1: The RLHF objective.

RLHF maximizes: $\max_{\pi_\theta} \mathbb{E}_{x \sim D, y \sim \pi_\theta}[r(x, y)] - \beta \cdot \text{KL}[\pi_\theta(y|x) \| \pi_{ref}(y|x)]$

Where $r(x,y)$ is the reward, $\pi_{ref}$ is the SFT reference policy, and $\beta$ controls the KL penalty strength.

Step 2: The optimal policy has a closed form.

The optimal policy for this objective (treating it as a KL-constrained optimization) is: $\pi^*(y|x) = \frac{\pi_{ref}(y|x) \exp(r(x,y)/\beta)}{Z(x)}$

Where $Z(x) = \sum_y \pi_{ref}(y|x) \exp(r(x,y)/\beta)$ is the partition function (normalizing constant).

Step 3: Rearrange to express the reward as a function of the policy.

Solving for $r(x,y)$: $r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$

Step 4: Substitute into the Bradley-Terry preference model.

The probability that $y_w$ is preferred over $y_l$: $P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))$

Substituting the reward expression - the $Z(x)$ terms cancel: $P(y_w \succ y_l | x) = \sigma\left(\beta \log \frac{\pi^*(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi^*(y_l|x)}{\pi_{ref}(y_l|x)}\right)$

Step 5: The DPO loss.

Treating $\pi_\theta$ as our model (approximating $\pi^*$), the maximum likelihood objective over observed preferences:

$\mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right]$

This is it. No reward model. No PPO. Just a binary cross-entropy loss over preference pairs, where the "score" for each response is how much more likely the trainable model produces it relative to the reference model.

Intuitive interpretation: DPO trains the model to assign higher probability to preferred responses relative to the reference model, and lower probability to dispreferred responses relative to the reference model. The relative increase/decrease is what matters, not the absolute probabilities.

Understanding the β Parameter

The $\beta$ parameter in DPO controls how much the policy can deviate from the reference model:

• High $\beta$ (e.g., 0.5-1.0): the model stays close to the reference. Preference optimization is conservative - only large preference signals cause meaningful policy updates.
• Low $\beta$ (e.g., 0.01-0.1): the model can deviate significantly from the reference. Preference optimization is aggressive - even weak preference signals cause large updates.

In practice, $\beta = 0.1$ is a common starting point. The paper found that $\beta$ values in the range 0.1 to 0.5 worked well across tasks.

note

β in DPO vs β in RLHF In RLHF, the KL coefficient $\beta$ is often called the "KL penalty coefficient." In DPO, $\beta$ plays the same role mathematically, but its interpretation differs: in DPO, $\beta$ modulates how strongly the model updates its policy relative to the reference. A larger $\beta$ means the log-ratio terms must be larger to produce the same gradient magnitude - effectively making the model more conservative.

How DPO Relates to the Reward Model

Here is the key insight that gives DPO its name: the DPO policy implicitly represents a reward model. From the derivation above:

$\hat{r}(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$

You can always compute an implicit reward for any response by comparing how much more likely the trained DPO model is to generate it versus the reference model. This implicit reward is exactly what DPO is optimizing - without ever explicitly instantiating it as a separate network.

This means: after DPO training, you can extract the implicit reward for any response. This is useful for evaluation, for identifying what the model considers high-quality, and for debugging.

DPO vs RLHF: Practical Quality Comparison

The honest answer: they produce comparable results on most tasks, with each having advantages.

DPO advantages:

• Simpler implementation (10x less code)
• More stable training (no PPO instability)
• Lower memory (2 models instead of 3)
• Easier hyperparameter tuning (only $\beta$ and learning rate vs many PPO parameters)

RLHF (PPO) advantages:

• Online learning: PPO can generate new responses and label them in a loop, continuously updating the reward signal. DPO uses a fixed offline dataset - the training set must be collected before training.
• For complex reasoning tasks with ground-truth feedback (code execution, math verification), PPO can incorporate that feedback in training. DPO cannot - it requires pre-collected preference pairs.
• Some studies show PPO slightly outperforms DPO on challenging reasoning benchmarks (but this is task-dependent and actively debated).

Empirical results from the DPO paper: on TL;DR summarization, DPO-trained GPT-2 XL was preferred over PPO-trained in 42% vs 34% of comparisons (rest were ties). On Anthropic HH-RLHF dialogue, DPO was preferred 57% vs 43%. DPO clearly worked.

Preference Datasets

DPO requires a dataset of triplets: (prompt, chosen response, rejected response). Several public datasets exist:

Anthropic HH-RLHF (Bai et al., 2022): 169K preference comparisons for helpfulness and harmlessness. Human labelers selected the more helpful/less harmful response from pairs of Claude-generated outputs. One of the most widely used DPO datasets.

OpenAI Summarize from Feedback (Stiennon et al., 2020): 93K preference comparisons for news article summarization. Human labelers chose the better summary.

HelpSteer (Wang et al., 2023): 37K (prompt, response, ratings) where ratings cover helpfulness, correctness, coherence, complexity, and verbosity. Useful for creating nuanced preference pairs.

UltraFeedback (Cui et al., 2023): 64K prompts with 4 responses each, rated by GPT-4 on instruction-following, truthfulness, honesty, and helpfulness. Commonly used for open-source DPO training.

Code: DPO Training with TRL

"""
DPO training with TRL DPOTrainer.
Complete pipeline: data preparation → training → evaluation.
"""

import torch
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig


# ---- Data format ----
# DPO requires triplets: (prompt, chosen, rejected)
# prompt: the input instruction
# chosen: the preferred response
# rejected: the less-preferred response


def prepare_dpo_dataset(raw_data: list) -> Dataset:
    """
    Convert raw preference data to DPO format.
    raw_data: list of dicts with keys: prompt, chosen, rejected
    """
    formatted = []
    for example in raw_data:
        formatted.append({
            "prompt": example["prompt"],
            "chosen": example["chosen"],
            "rejected": example["rejected"],
        })
    return Dataset.from_list(formatted)


# Example data
sample_data = [
    {
        "prompt": "What is the capital of France?",
        "chosen": "The capital of France is Paris.",
        "rejected": (
            "France is a beautiful country in Western Europe. "
            "It has many wonderful cities including Lyon, Marseille, and Paris. "
            "Paris is the capital and largest city, known for the Eiffel Tower."
        ),
        # Rejected: technically correct but verbose and unfocused
    },
    {
        "prompt": "Explain backpropagation in one sentence.",
        "chosen": (
            "Backpropagation computes gradients of the loss with respect to model "
            "parameters by applying the chain rule from output to input layers."
        ),
        "rejected": (
            "I don't know, backpropagation is complicated and involves calculus. "
            "You should look it up."
        ),
    },
    {
        "prompt": "Write a Python function to reverse a string.",
        "chosen": (
            "```python\ndef reverse_string(s: str) -> str:\n"
            "    return s[::-1]\n```"
        ),
        "rejected": (
            "You can reverse a string using a for loop: "
            "```python\ndef reverse_string(s):\n"
            "    result = ''\n"
            "    for char in s:\n"
            "        result = char + result\n"
            "    return result\n```"
        ),
        # Rejected: works but less Pythonic
    },
]


def run_dpo_training(
    sft_model_name: str,
    output_dir: str = "./dpo-model",
    beta: float = 0.1,
    train_data: list = None,
):
    """
    Full DPO training pipeline.

    sft_model_name: the fine-tuned model to start from
                    (must already be instruction-tuned with SFT)
    beta: temperature parameter - higher = stay closer to reference
    """
    tokenizer = AutoTokenizer.from_pretrained(sft_model_name)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token

    # Policy model (trainable)
    model = AutoModelForCausalLM.from_pretrained(
        sft_model_name,
        torch_dtype=torch.bfloat16,
        use_cache=False,
    )

    # Reference model (frozen SFT model)
    # TRL DPOTrainer creates the reference model automatically from the same checkpoint
    # Alternatively, load explicitly:
    # ref_model = AutoModelForCausalLM.from_pretrained(
    #     sft_model_name, torch_dtype=torch.bfloat16
    # )

    # Prepare dataset
    if train_data is None:
        train_data = sample_data
    dataset = prepare_dpo_dataset(train_data)

    # Split into train/eval
    dataset = dataset.train_test_split(test_size=0.1)

    # DPO Configuration
    dpo_config = DPOConfig(
        output_dir=output_dir,
        num_train_epochs=1,           # DPO typically needs only 1-3 epochs
        per_device_train_batch_size=4,
        per_device_eval_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=5e-7,           # DPO uses MUCH lower LR than SFT (5e-7 to 1e-6)
        lr_scheduler_type="cosine",
        warmup_ratio=0.1,
        bf16=True,
        beta=beta,                    # KL regularization coefficient
        max_length=1024,              # Max length of prompt + chosen/rejected
        max_prompt_length=512,        # Max prompt length
        evaluation_strategy="steps",
        eval_steps=100,
        logging_steps=10,
        save_steps=200,
        report_to="none",
    )

    trainer = DPOTrainer(
        model=model,
        ref_model=None,               # None = TRL uses model copy as reference
        args=dpo_config,
        train_dataset=dataset["train"],
        eval_dataset=dataset["test"],
        processing_class=tokenizer,
    )

    trainer.train()
    trainer.save_model(output_dir)
    tokenizer.save_pretrained(output_dir)

    return trainer


# ---- Extracting the implicit reward ----

def compute_implicit_reward(
    prompt: str,
    response: str,
    dpo_model,
    ref_model,
    tokenizer,
    beta: float = 0.1,
) -> float:
    """
    Compute the implicit reward r(x,y) = beta * log(pi_dpo(y|x) / pi_ref(y|x))

    This is the reward that DPO implicitly learned.
    Higher score = DPO model considers this response more preferred.
    """
    def get_log_prob(model, prompt, response):
        """Compute log probability of response given prompt."""
        text = prompt + response
        inputs = tokenizer(text, return_tensors="pt")
        prompt_len = len(tokenizer(prompt)["input_ids"])

        with torch.no_grad():
            outputs = model(**inputs, labels=inputs["input_ids"])

        # Get per-token log probs
        logits = outputs.logits[:, :-1, :]  # (1, seq_len-1, vocab)
        labels = inputs["input_ids"][:, 1:]  # (1, seq_len-1)

        log_probs = torch.log_softmax(logits, dim=-1)
        token_log_probs = log_probs.gather(
            -1, labels.unsqueeze(-1)
        ).squeeze(-1)  # (1, seq_len-1)

        # Sum log probs over response tokens only
        response_log_prob = token_log_probs[0, prompt_len-1:].sum().item()
        return response_log_prob

    log_prob_dpo = get_log_prob(dpo_model, prompt, response)
    log_prob_ref = get_log_prob(ref_model, prompt, response)

    implicit_reward = beta * (log_prob_dpo - log_prob_ref)
    return implicit_reward


# ---- DPO with LoRA (memory-efficient) ----

def run_dpo_with_lora(
    sft_model_name: str,
    output_dir: str = "./dpo-lora",
    beta: float = 0.1,
):
    """
    DPO with LoRA for memory-efficient alignment.
    Useful for aligning large models (13B+) with limited GPU.
    """
    from peft import LoraConfig, get_peft_model, TaskType

    tokenizer = AutoTokenizer.from_pretrained(sft_model_name)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token

    model = AutoModelForCausalLM.from_pretrained(
        sft_model_name,
        torch_dtype=torch.bfloat16,
        use_cache=False,
    )

    # Apply LoRA - only train adapter matrices for DPO
    lora_config = LoraConfig(
        r=64,
        lora_alpha=64,
        target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
        task_type=TaskType.CAUSAL_LM,
    )
    model = get_peft_model(model, lora_config)
    model.print_trainable_parameters()

    dataset = prepare_dpo_dataset(sample_data)

    dpo_config = DPOConfig(
        output_dir=output_dir,
        num_train_epochs=1,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        learning_rate=5e-7,
        bf16=True,
        beta=beta,
        max_length=1024,
        max_prompt_length=512,
    )

    trainer = DPOTrainer(
        model=model,
        ref_model=None,  # With PEFT, TRL automatically disables LoRA in reference
        args=dpo_config,
        train_dataset=dataset,
        processing_class=tokenizer,
    )

    trainer.train()
    return trainer

DPO Extensions and Variants

IPO: Identity Preference Optimization (Azar et al., 2023)

IPO identified a theoretical issue with DPO: the DPO loss can be driven to zero even when the model perfectly fits the preference data, by making the log-ratio terms arbitrarily large. IPO adds a regularization term that prevents this:

$\mathcal{L}_{IPO} = \mathbb{E}\left[\left(\log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} - \frac{1}{2\beta}\right)^2\right]$

IPO is more stable when preferences are deterministic (clear winner every time) and the model tends to overfit.

KTO: Kahneman-Tversky Optimization (Ethayarajh et al., 2024)

KTO does not require preference pairs at all - just binary feedback (this response was good / this response was bad). Inspired by Kahneman-Tversky prospect theory, KTO models human decision-making under uncertainty. Useful when you have labels for individual responses rather than comparative pairs.

SimPO: Simple Preference Optimization (Meng et al., 2024)

SimPO removes the reference model entirely. Instead of comparing to $\pi_{ref}$, it normalizes by the length of the response. This eliminates the need to load a frozen reference model, halving the memory requirement. Strong empirical results on instruction-following benchmarks.

$\mathcal{L}_{SimPO} = -\log \sigma\left(\frac{\beta}{|y_w|}\log \pi_\theta(y_w|x) - \frac{\beta}{|y_l|}\log \pi_\theta(y_l|x) - \gamma\right)$

ORPO: Odds Ratio Preference Optimization (Hong et al., 2024)

ORPO combines SFT and DPO into a single training objective, eliminating the need for a separate SFT phase. The loss is the standard language modeling loss plus an odds ratio term that penalizes rejected responses:

$\mathcal{L}_{ORPO} = \mathcal{L}_{SFT} + \lambda \cdot \mathcal{L}_{OR}$

This simplifies the training pipeline to a single stage: no SFT first, then DPO. Train once, get alignment. Increasingly popular for its simplicity.

Production Engineering Notes

DPO Learning Rate is Much Lower Than SFT

A critical point: DPO uses learning rates in the range of 5e-7 to 1e-6, which is 10-50x lower than typical SFT learning rates (2e-5). This is because DPO is making small adjustments to an already instruction-tuned model - you want to shift preferences without destroying the SFT training. Using the SFT learning rate for DPO will cause the model to deviate too far from the reference and produce degraded outputs.

The Chosen and Rejected Responses Must Come from the Reference Model

An important subtlety: the DPO objective is derived assuming the chosen and rejected responses were sampled from the reference model (the SFT model). If your preference data was collected using a different model (e.g., GPT-4 outputs as "chosen" and base model outputs as "rejected"), the theoretical justification breaks down. In practice, DPO still works reasonably well with such data, but the best results come when the preference pairs are generated by the model being aligned (the SFT model).

Evaluating DPO Quality

Evaluating DPO is challenging because the training data is preference pairs and the evaluation task is usually open-ended generation. Use:

1. Win rate vs SFT baseline: using GPT-4 or another strong model as judge, compare DPO model outputs vs SFT model outputs. Target win rate greater than 60%.
2. MT-Bench: standardized instruction following evaluation.
3. AlpacaEval 2: win rate against GPT-4-turbo as reference on 805 instructions.
4. Implicit reward distribution: plot the implicit reward $\beta \log(\pi_\theta/\pi_{ref})$ for chosen vs rejected responses in a held-out test set. They should be well-separated.

Common Mistakes

danger

Not starting from an SFT-trained model DPO fine-tunes an already-instruction-tuned model (SFT model) to align preferences. If you apply DPO to a base model that has not been SFT-trained, the reference model distribution is the base model's distribution, and the preference pairs have no meaningful relationship to the base model's outputs. The training will be unstable and the results will be poor. Always SFT first, then DPO.

danger

Using the wrong learning rate (SFT learning rate for DPO) DPO requires a much lower learning rate than SFT - typically 5e-7 to 1e-6, versus 2e-5 for SFT. Using an SFT-scale learning rate for DPO causes the model to deviate significantly from the reference distribution, defeating the purpose of the KL penalty term and often producing a model that is worse than the SFT baseline. If your DPO model performs worse than the SFT model it was trained from, check the learning rate first.

warning

Training for too many epochs on a small preference dataset With 1,000-10,000 preference pairs, training for more than 1-2 epochs causes overfitting. The model memorizes the specific chosen responses (rather than learning the general preference pattern) and produces repetitive outputs. Watch the implicit reward distribution during training: if the margin between chosen and rejected rewards becomes very large (greater than 10) while train loss is near zero, you are overfitting.

tip

Use DPO with LoRA to save memory DPO requires two models in memory: the trainable policy and the frozen reference model. For 7B+ models, this can exceed single-GPU memory. When using PEFT/LoRA with DPO, TRL's DPOTrainer automatically handles the reference model: it disables the LoRA adapters for the reference model's forward pass, so the frozen base model serves as the reference without requiring a separate model copy. This halves the memory requirement for DPO training.

Interview Q&A

Q1: What is the key mathematical insight behind DPO that makes it work without a reward model?

DPO derives from a closed-form solution to the KL-constrained RLHF objective. For the optimization problem "maximize expected reward minus KL divergence from reference," the optimal policy is $\pi^*(y|x) \propto \pi_{ref}(y|x) \exp(r(x,y)/\beta)$. Rearranging, the reward can be expressed as $r(x,y) = \beta \log \pi^*(y|x)/\pi_{ref}(y|x) + \text{const}$. Substituting this into the Bradley-Terry preference probability $P(y_w \succ y_l) = \sigma(r_w - r_l)$, the constant cancels and you get a loss that depends only on the policy ratio $\pi_\theta/\pi_{ref}$ - no separate reward model needed. The DPO policy implicitly represents a reward model.

Q2: Why does DPO need a reference model if there is no reward model?

The reference model (frozen SFT model) serves as the "anchor" in DPO's log-ratio term: $\log \pi_\theta(y|x)/\pi_{ref}(y|x)$. This ratio measures how much the trainable model deviates from the reference for a given response. Without this ratio, DPO would just maximize the probability of chosen responses and minimize the probability of rejected responses - which would collapse to language modeling without the preference aspect. The reference model ensures that the model is rewarded not for having high absolute probability for chosen responses, but for having higher probability for chosen responses relative to what the reference model would predict.

Q3: How does the beta parameter affect DPO training?

$\beta$ controls the strength of the KL regularization. High $\beta$ (e.g., 0.5): the policy stays close to the reference. The log-ratio terms must be large to produce meaningful gradient updates - only strong preference signals matter. The model makes small, conservative adjustments. Low $\beta$ (e.g., 0.01): the policy can deviate significantly from the reference. Small log-ratio differences produce large gradient updates. The model can change more aggressively to fit preferences. Typical starting point: $\beta = 0.1$. If the model drifts too far from the SFT baseline (quality drops on general tasks), increase $\beta$. If the preference optimization is too slow, decrease $\beta$.

Q4: What is the difference between DPO and IPO, and when would you choose IPO?

DPO maximizes $\log \sigma(\text{log-ratio margin})$ - the sigmoid can become saturated when preferences are very clear, leading to near-zero gradient when the model already correctly assigns higher probability to chosen over rejected. IPO fixes this by minimizing the squared distance between the log-ratio margin and $1/(2\beta)$, preventing saturation. Choose IPO when: your preference data has very clear, consistent preferences (where DPO tends to overfit), or when you observe that DPO training loss drops to near zero very quickly but model quality plateaus. In practice, DPO and IPO perform similarly for most tasks - DPO is more commonly used due to its wider adoption and simpler implementation.

Q5: Compare DPO and PPO (RLHF) in terms of practical use cases where each excels.

PPO excels when: you need online learning (generate responses → label → train in a loop), the reward signal comes from execution (code testing, math verification), or you want to combine multiple reward signals (helpfulness + safety + factuality simultaneously with different weights). DPO excels when: you have a fixed offline preference dataset, you want simple and stable training, you have memory constraints (only 2 models instead of 3), or you want fast iteration. In practice, for most instruction-following and chat alignment tasks, DPO produces comparable results to PPO with much less engineering complexity. For complex reasoning tasks with verifiable ground truth, PPO (or variants like GRPO) often wins because it can incorporate execution feedback.

Advanced: Building High-Quality DPO Datasets

The quality of DPO training is determined almost entirely by the quality of your preference dataset. Here is a practical guide to building one:

Strategy 1: Use an Existing Public Dataset

The fastest path to DPO training. Recommended datasets:

"""Load and format public DPO datasets for TRL DPOTrainer."""

from datasets import load_dataset


def load_anthropic_hh(split="train", max_examples=10000):
    """
    Anthropic HH-RLHF dataset: 169K examples of helpful + harmless preferences.
    Human raters chose the more helpful/harmless response from Claude pairs.
    """
    dataset = load_dataset("Anthropic/hh-rlhf", split=split)

    def format_hh(example):
        return {
            "prompt": example["chosen"].rsplit("Assistant:", 1)[0] + "Assistant:",
            "chosen": example["chosen"].rsplit("Assistant:", 1)[1].strip(),
            "rejected": example["rejected"].rsplit("Assistant:", 1)[1].strip(),
        }

    return dataset.map(format_hh).select(range(min(max_examples, len(dataset))))


def load_ultrafeedback(split="train", max_examples=20000):
    """
    UltraFeedback: 64K prompts with GPT-4 ratings on 4 axes.
    Create preference pairs from highest vs lowest rated responses.
    """
    dataset = load_dataset("openbmb/UltraFeedback", split=split)

    def format_ultrafeedback(example):
        # Get responses sorted by overall score
        completions = example["completions"]
        scored = [
            (c["response"], c.get("overall_score", 0))
            for c in completions
        ]
        scored.sort(key=lambda x: x[1], reverse=True)

        if len(scored) < 2:
            return None

        return {
            "prompt": example["instruction"],
            "chosen": scored[0][0],   # Highest-scored response
            "rejected": scored[-1][0], # Lowest-scored response
        }

    return (
        dataset
        .map(format_ultrafeedback)
        .filter(lambda x: x is not None)
        .select(range(min(max_examples, len(dataset))))
    )

Strategy 2: Generate Preferences with a Judge Model

When you have a collection of prompts and want to create preferences without human annotation:

"""
AI-generated preference dataset using a judge LLM.
Works well for instruction-following and factual tasks.
"""

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import json


JUDGE_PROMPT = """You are evaluating two responses to a user request.

User request: {prompt}

Response A:
{response_a}

Response B:
{response_b}

Which response is better? Consider:
- Helpfulness: Does it fully address the request?
- Accuracy: Is the information correct?
- Clarity: Is it well-written and easy to understand?
- Safety: Does it avoid harmful content?

Answer with ONLY "A" or "B" followed by a one-sentence reason.
"""


def judge_responses_with_llm(
    prompt: str,
    response_a: str,
    response_b: str,
    judge_model,
    judge_tokenizer,
) -> str:
    """Use LLM as judge to select the better response. Returns 'A' or 'B'."""
    judge_input = JUDGE_PROMPT.format(
        prompt=prompt,
        response_a=response_a,
        response_b=response_b,
    )

    inputs = judge_tokenizer(judge_input, return_tensors="pt")

    with torch.no_grad():
        output = judge_model.generate(
            **inputs,
            max_new_tokens=50,
            temperature=0.1,         # Low temp for deterministic judgment
            do_sample=True,
            pad_token_id=judge_tokenizer.eos_token_id,
        )

    response_text = judge_tokenizer.decode(
        output[0][inputs["input_ids"].shape[1]:],
        skip_special_tokens=True,
    ).strip()

    # Extract A or B from response
    if response_text.startswith("A"):
        return "A"
    elif response_text.startswith("B"):
        return "B"
    else:
        return None  # Could not determine - skip this pair


def build_preference_dataset_with_judge(
    prompts: list,
    policy_model,
    policy_tokenizer,
    judge_model,
    judge_tokenizer,
    num_responses_per_prompt: int = 4,
) -> list:
    """
    Build DPO dataset by generating multiple responses per prompt
    and using LLM judge to select chosen/rejected pairs.
    """
    preference_pairs = []

    for prompt in prompts:
        # Generate multiple responses with different temperatures
        inputs = policy_tokenizer(prompt, return_tensors="pt")
        responses = []

        for temp in [0.5, 0.7, 0.9, 1.1][:num_responses_per_prompt]:
            with torch.no_grad():
                output = policy_model.generate(
                    **inputs,
                    max_new_tokens=256,
                    temperature=temp,
                    do_sample=True,
                    pad_token_id=policy_tokenizer.eos_token_id,
                )
            response = policy_tokenizer.decode(
                output[0][inputs["input_ids"].shape[1]:],
                skip_special_tokens=True,
            )
            responses.append(response)

        # Compare pairs and select the best
        best_response = None
        worst_response = None
        best_wins = 0

        for i, r_a in enumerate(responses):
            wins = 0
            for j, r_b in enumerate(responses):
                if i == j:
                    continue
                winner = judge_responses_with_llm(
                    prompt, r_a, r_b, judge_model, judge_tokenizer
                )
                if winner == "A":
                    wins += 1

            if best_response is None or wins > best_wins:
                best_wins = wins
                best_response = r_a
            if worst_response is None or wins < best_wins:
                worst_response = r_a

        if best_response and worst_response and best_response != worst_response:
            preference_pairs.append({
                "prompt": prompt,
                "chosen": best_response,
                "rejected": worst_response,
            })

    return preference_pairs

DPO Debugging Checklist

When DPO training does not improve over the SFT baseline, diagnose systematically:

def diagnose_dpo_training(trainer, eval_dataset, tokenizer):
    """
    Diagnostic checks for DPO training issues.
    Run after each training epoch.
    """
    issues = []

    # Check 1: Is the model actually updating?
    # Compare output probabilities at start vs end of training
    # If chosen/rejected rewards are not separating, training is not working

    # Check 2: Learning rate
    # DPO LR should be 5e-7 to 1e-6, NOT 2e-5 (SFT range)
    current_lr = trainer.optimizer.param_groups[0]["lr"]
    if current_lr > 5e-6:
        issues.append(f"WARNING: Learning rate {current_lr:.2e} is too high for DPO. "
                      f"Use 5e-7 to 1e-6.")

    # Check 3: Beta value
    beta = trainer.args.beta
    if beta < 0.01:
        issues.append(f"WARNING: Beta {beta} is very low - policy may drift far from SFT.")
    elif beta > 1.0:
        issues.append(f"WARNING: Beta {beta} is very high - DPO updates will be negligible.")

    # Check 4: Dataset format
    sample = eval_dataset[0]
    if "prompt" not in sample or "chosen" not in sample or "rejected" not in sample:
        issues.append("ERROR: Dataset missing required fields: prompt, chosen, rejected")

    # Check 5: Response length balance
    chosen_lens = [len(tokenizer(s["chosen"])["input_ids"]) for s in eval_dataset]
    rejected_lens = [len(tokenizer(s["rejected"])["input_ids"]) for s in eval_dataset]

    avg_chosen_len = sum(chosen_lens) / len(chosen_lens)
    avg_rejected_len = sum(rejected_lens) / len(rejected_lens)

    if avg_chosen_len > avg_rejected_len * 1.5:
        issues.append(
            f"WARNING: Chosen responses are {avg_chosen_len:.0f} tokens avg vs "
            f"{avg_rejected_len:.0f} for rejected. Model may learn to prefer length."
        )

    if issues:
        for issue in issues:
            print(issue)
    else:
        print("No DPO configuration issues detected.")

    return issues

Practical Guidance: SFT + DPO vs SFT + RLHF in 2025

As of 2025, the consensus from the open-source community and industry practitioners:

For production instruction-following models (7B-70B):

• SFT on 50K-500K high-quality instruction examples
• DPO on 20K-100K preference pairs (UltraFeedback + domain-specific)
• This pipeline produces models competitive with GPT-3.5-class on most tasks
• Total training time on 8x A100s: 2-3 days

When to add RLHF:

• Reasoning tasks with verifiable outcomes (math, code)
• Safety-critical deployments where reward hacking must be carefully monitored
• When you have the infrastructure for online data collection

The verdict on DPO vs RLHF: DPO has largely replaced PPO-based RLHF for offline preference datasets. The gap in quality is small (often less than 2% on benchmarks) while the engineering complexity reduction is enormous. Most state-of-the-art open models released in 2024-2025 use DPO or a DPO variant (ORPO, SimPO) rather than PPO.

note

GRPO: The New Challenger Group Relative Policy Optimization (GRPO, Shao et al., 2024) emerged from DeepSeek as a PPO variant specifically designed for reasoning. Instead of a learned reward model, GRPO uses verifiable rewards (code execution, math checking). It generates multiple responses per prompt and uses them to estimate advantage values without a separate value function. DeepSeek-R1's extraordinary mathematical reasoning is largely attributed to GRPO training. For reasoning tasks with ground-truth verification, GRPO is rapidly becoming the method of choice.

---

DPO Extensions in Depth

ORPO - Odds Ratio Preference Optimization

ORPO (Hong et al., 2024) is a single-stage alternative that eliminates the need for a separate SFT phase. It adds an odds ratio penalty directly into the SFT loss:

$\mathcal{L}_{ORPO} = \mathcal{L}_{SFT} + \lambda \cdot \mathbb{E}\left[\log \sigma\left(\log \frac{\pi_\theta(y_w|x)}{1 - \pi_\theta(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{1 - \pi_\theta(y_l|x)}\right)\right]$

The odds ratio $\frac{p}{1-p}$ measures how much the model prefers generating $y_w$ over any alternative. The ORPO loss penalizes the model for giving high odds to $y_l$ while rewarding high odds for $y_w$. Since it operates on the per-token probabilities rather than the full-sequence probability ratios (DPO), it is simpler to implement and doesn't require a reference model.

from trl import ORPOConfig, ORPOTrainer

def run_orpo_training(
    model_name: str,
    preference_dataset,    # Dataset with "prompt", "chosen", "rejected" columns
    output_dir: str,
) -> str:
    """Single-stage ORPO training - no separate SFT phase needed."""
    from transformers import AutoModelForCausalLM, AutoTokenizer

    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.bfloat16,
        attn_implementation="flash_attention_2",
    )
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token

    orpo_config = ORPOConfig(
        output_dir=output_dir,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        num_train_epochs=2,
        learning_rate=8e-6,      # ORPO typically uses lower LR than DPO
        lambda_=0.1,             # Weight of the odds ratio penalty term
        max_length=2048,
        max_prompt_length=512,
        bf16=True,
        logging_steps=25,
        save_strategy="epoch",
    )

    trainer = ORPOTrainer(
        model=model,
        args=orpo_config,
        train_dataset=preference_dataset,
        tokenizer=tokenizer,
    )
    trainer.train()
    trainer.save_model(output_dir)
    return output_dir

SimPO - Simple Preference Optimization

SimPO (Meng et al., 2024) makes two key changes relative to DPO: (1) it uses sequence-average log probability (normalizing by sequence length) rather than sum, and (2) it adds a target reward margin $\gamma$ to ensure the winning response has a strictly higher implicit reward than the losing response by at least $\gamma$:

$\mathcal{L}_{SimPO} = -\log \sigma\left(\frac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \gamma\right)$

The length normalization prevents a known DPO failure mode where the model learns to prefer longer responses regardless of quality (longer responses have lower absolute log probability, which can confuse DPO). SimPO also does not require a reference model, simplifying training.

import torch
import torch.nn.functional as F

def simpo_loss(
    policy_logps_chosen: torch.Tensor,     # Log probs of chosen responses (per token)
    policy_logps_rejected: torch.Tensor,   # Log probs of rejected responses (per token)
    chosen_lengths: torch.Tensor,          # Number of tokens in each chosen response
    rejected_lengths: torch.Tensor,        # Number of tokens in each rejected response
    beta: float = 2.5,
    gamma: float = 1.0,
) -> torch.Tensor:
    """
    SimPO loss - length-normalized, no reference model required.
    policy_logps: summed log probs over the sequence (shape: batch_size)
    lengths: number of tokens in the response (shape: batch_size)
    """
    # Length-normalized log probabilities
    chosen_avg_logps = policy_logps_chosen / chosen_lengths.float()
    rejected_avg_logps = policy_logps_rejected / rejected_lengths.float()

    # Reward difference with margin
    reward_diff = beta * (chosen_avg_logps - rejected_avg_logps) - gamma

    loss = -F.logsigmoid(reward_diff).mean()

    # Metrics
    with torch.no_grad():
        accuracy = (reward_diff > 0).float().mean()

    return loss, accuracy

---

DPO in Production: Practical Considerations

Dataset Size and Quality Trade-offs

import random

def create_tiered_preference_dataset(
    raw_pairs: list[dict],
    quality_tiers: dict[str, float] = None,
) -> list[dict]:
    """
    Organize preference pairs into quality tiers and sample accordingly.
    Higher-quality pairs contribute more to training.

    quality_tiers: {tier_name: sampling_weight}
    """
    if quality_tiers is None:
        quality_tiers = {
            "high":   2.0,    # Human-verified pairs - sample 2x
            "medium": 1.0,    # LLM-judged pairs - sample 1x
            "low":    0.3,    # Weak signal pairs - undersample
        }

    tiered = {tier: [] for tier in quality_tiers}
    for pair in raw_pairs:
        tier = pair.get("quality_tier", "medium")
        if tier in tiered:
            tiered[tier].append(pair)

    print("Dataset composition:")
    for tier, pairs in tiered.items():
        print(f"  {tier}: {len(pairs)} pairs (weight: {quality_tiers[tier]})")

    # Weighted sampling to create final dataset
    final_dataset = []
    for tier, pairs in tiered.items():
        weight = quality_tiers[tier]
        n_samples = int(len(pairs) * weight)
        if n_samples > len(pairs):
            # Oversample with replacement
            sampled = random.choices(pairs, k=n_samples)
        else:
            # Undersample without replacement
            sampled = random.sample(pairs, n_samples)
        final_dataset.extend(sampled)

    random.shuffle(final_dataset)
    print(f"Final dataset size: {len(final_dataset)} pairs")
    return final_dataset


def estimate_dpo_training_cost(
    model_size_b: float,
    dataset_size: int,
    num_epochs: int,
    hardware: str = "a100_80gb",
) -> dict:
    """
    Estimate DPO training cost (time and cloud cost).
    """
    # DPO requires 2 forward passes (policy + reference) per batch step
    # Memory requirement: 2x policy model weight size + optimizer state (LoRA)
    hardware_specs = {
        "a100_80gb": {"vram_gb": 80, "tflops": 312, "cost_per_hour": 3.0},
        "h100_80gb": {"vram_gb": 80, "tflops": 989, "cost_per_hour": 8.0},
        "rtx_4090": {"vram_gb": 24, "tflops": 165, "cost_per_hour": 0.4},
    }
    hw = hardware_specs.get(hardware, hardware_specs["a100_80gb"])

    # Approximate: 1 DPO step ≈ 2x SFT step compute (two forward passes)
    # Tokens processed = dataset_size * avg_sequence_length * 2 (chosen + rejected)
    avg_seq_len = 1024
    total_tokens = dataset_size * avg_seq_len * 2 * num_epochs

    # FLOPs ≈ 6 * N * T (forward + backward)
    total_flops = 6 * model_size_b * 1e9 * total_tokens

    # Time estimate with 40% MFU on chosen hardware
    training_hours = total_flops / (0.4 * hw["tflops"] * 1e12 * 3600)

    return {
        "estimated_hours": f"{training_hours:.1f}h",
        "estimated_cost": f"${training_hours * hw['cost_per_hour']:.2f}",
        "hardware": hardware,
        "total_tokens_processed": f"{total_tokens/1e9:.2f}B",
        "note": "Assumes LoRA DPO (ref model is frozen). Full DPO requires ~2x memory."
    }

# Example: DPO on 7B model, 50K pairs, 1 epoch, single A100
print(estimate_dpo_training_cost(7, 50_000, 1, "a100_80gb"))

---

Key Takeaways

DPO is one of the most impactful algorithmic insights in recent LLM research. By deriving a closed-form loss that directly encodes the RLHF objective without an explicit reward model or RL loop, it collapsed a three-phase training pipeline into a single fine-tuning step.

The mathematical elegance is matched by practical utility: DPO produces models competitive with PPO-based RLHF on most instruction-following benchmarks, trains 3–10x faster, and requires no specialized RL infrastructure. The HuggingFace TRL library makes DPO accessible in under 50 lines of configuration code.

As of 2025, DPO or a DPO variant (ORPO, SimPO, KTO) is the default alignment training method for open-source models. The RLHF → DPO transition mirrors the earlier SGD → Adam transition: a theoretically grounded simplification that works so well in practice that the older, more complex approach is now rarely used for its original purpose.

:::tip 🎮 Interactive Playground

Visualize this concept: Try the DPO vs RLHF demo on the EngineersOfAI Playground - no code required.

:::