RLHF & Alignment

From "predict the next token" to "do what humans actually want."

7 papers~11–14 hours readingChronological

Alignment is the problem of making language models do what we want, not just what they were trained on. The field moved from reward models and PPO (computationally expensive, unstable) through Constitutional AI (scalable self-critique) to preference optimization methods like DPO and ORPO that don't need reward models at all. Each paper is a step toward alignment methods that are cheaper, more stable, and more scalable.

Read in order. Each paper fixes a failure mode exposed by the previous one. Click any card to collapse it.

1

Fixes: Models optimising for metrics that don't match human preferences

2020 · NeurIPS 2020▼

Learning to Summarize — RLHF Origin

Stiennon, Ouyang, Wu et al. (OpenAI)

Learning to Summarize with Human Feedback

First systematic application of RLHF to language models. Trains a reward model on human preference comparisons (which summary is better?), then uses PPO to fine-tune the policy to maximise reward. Outperforms supervised fine-tuning on human evaluations of summary quality.

Key insight: You can't fully specify what 'good' looks like in a loss function. Human preference data, even in small amounts, captures nuance that automatic metrics miss. This paper proves the idea works at meaningful scale — everything after is refinement.

TPU clusterMonthsWith caveats

arXiv →

2

Fixes: Pretrained LLMs optimising for next-token prediction instead of helpfulness

2022 · NeurIPS 2022▼

InstructGPT — RLHF at Scale

Ouyang, Wu, Jiang et al. (OpenAI)

Training Language Models to Follow Instructions with Human Feedback

Applies RLHF to GPT-3 to create InstructGPT. Three-stage pipeline: supervised fine-tuning on demonstrations, reward model training on preference comparisons, PPO fine-tuning to maximise reward. InstructGPT (1.3B) is preferred over GPT-3 (175B) by human raters on most tasks.

Key insight: A smaller model trained with human feedback consistently beats a model 100x its size trained only on prediction. This single finding justified the entire RLHF research agenda and is the direct ancestor of ChatGPT and every instruction-following LLM since.

Engineering note: PPO is notoriously difficult to tune. The reward hacking problem — models learning to game the reward model rather than be genuinely helpful — is not fully solved here and remains an active research area.

TPU clusterMonthsWith caveats

arXiv →

3

Fixes: RLHF requiring expensive human preference labels at scale

2022 · arXiv 2022▼

Constitutional AI — Self-Critique & RLAIF

Bai, Jones, Ndousse et al. (Anthropic)

Constitutional AI: Harmlessness from AI Feedback

Introduces Constitutional AI (CAI): the model critiques and revises its own outputs according to a written 'constitution' of principles. Supervised learning on these self-revised outputs replaces some human preference data. RL from AI feedback (RLAIF) uses another LLM to generate preference labels at scale.

Key insight: You can encode values in natural language and have the model apply them to itself. This scales alignment annotation from human-speed to model-speed. RLAIF is now standard practice and directly enables Anthropic's Claude models.

TPU clusterWeeksWith caveats

arXiv →

4

Fixes: PPO's computational expense, instability, and reward hacking

2023 · NeurIPS 2023▼

DPO — RLHF Without a Reward Model

Rafailov, Sharma, Mitchell et al. (Stanford)

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Derives a closed-form mapping between a reward function and an optimal policy. This allows direct training on preference pairs (chosen vs rejected) using a simple classification loss — no separate reward model, no RL training loop, no PPO. Achieves comparable or better results at a fraction of the compute.

Key insight: The reward model is implicit in the policy. DPO doesn't eliminate the reward signal — it internalises it into the cross-entropy loss. This insight is elegant and practical: standard fine-tuning infrastructure works, no RL libraries needed.

Engineering note: DPO is sensitive to the quality and distribution of preference pairs. Noisy or out-of-distribution preference data degrades results more than in RLHF. Data curation matters more than in supervised fine-tuning.

Cloud GPUDaysShip it now

arXiv →

5

Fixes: DPO requiring a reference model and two-stage training

2024 · EMNLP 2024▼

ORPO — One-Stage Preference Optimization

Hong, Lee, Thorne (KAIST / Contextual AI)

ORPO: Monolithic Preference Optimization without Reference Model

Adds an odds ratio penalty directly to the SFT loss, enabling single-stage preference alignment without a reference model or separate SFT phase. The odds ratio term penalises the probability of rejected responses relative to chosen ones while training the model to follow instructions.

Key insight: Combining SFT and preference alignment into one stage with one objective is more computationally efficient than DPO and eliminates the reference model dependency. ORPO matches or beats DPO with 30-40% less compute. This is the current state-of-the-art for efficient preference tuning.

Consumer GPUDaysShip it now

arXiv →

6

Fixes: DPO's reward formulation favouring verbose responses

2024 · NeurIPS 2024▼

SimPO — Length-Normalised Preference Optimization

Meng, Xia, Chen (Princeton / Meta)

SimPO: Simple Preference Optimization with a Reference-Free Reward

Replaces DPO's implicit reward (log probability ratio with reference model) with a length-normalised average log probability. Adds a target reward margin to ensure chosen responses score above rejected ones by a minimum margin. No reference model needed.

Key insight: Length normalisation is a critical detail. DPO without it tends to reward verbosity — the model learns that longer responses are more likely to be 'chosen'. SimPO's normalisation corrects this bias, producing more concise and correct completions.

Consumer GPUDaysShip it now

arXiv →

7

Fixes: Alignment methods not scaling to frontier model capabilities

2024 · Various 2024 papers▼

Scalable RLHF — The Next Frontier

Multiple groups (OpenAI, Anthropic, DeepMind — ongoing)

Scaling Reinforcement Learning from Human Feedback

The frontier of alignment research in 2024-2025: process reward models (rewarding reasoning steps, not just final answers), debate as an alignment mechanism, scalable oversight (using AI to supervise AI on tasks too hard for humans to evaluate), and iterated amplification.

Key insight: As models get more capable, human evaluation becomes less reliable (we can't verify a model's proof of a hard theorem). The next phase of alignment is designing oversight mechanisms that scale with capability — not just preference data at training time.

Engineering note: This is active research. No single paper captures the current state. Follow the Alignment Forum and the research pages of OpenAI, Anthropic, and DeepMind for the latest.

TPU clusterMonthsResearch only

arXiv →

---

The Practical Takeaway

For most production use cases today:

Small dataset of good demonstrations → SFT
Preference pairs (chosen/rejected)   → ORPO or DPO
Evaluate with LLM judge              → iterate

Full RLHF with PPO is only necessary if you have a large human labelling budget and a specific safety requirement. For most applications, DPO or ORPO is faster, cheaper, and just as effective.

Related

• RAG Evolution Roadmap →
• Agents Evolution Roadmap →