RLHF Pipeline Visualization - Interactive Visualization

RLHF (Reinforcement Learning from Human Feedback) is the technique that transformed raw language models into aligned assistants. It has three stages: first, supervised fine-tuning on demonstration data; second, training a reward model on human preference comparisons; third, using PPO to optimize the language model against the reward model while a KL penalty prevents it from drifting too far from the SFT baseline.

• See Stage 1: SFT - the base LLM is fine-tuned on human-written demonstration responses to learn the basic task format
• See Stage 2: reward model - human annotators rank multiple responses; the RM is trained to predict these rankings
• See Stage 3: PPO loop - the LLM generates responses, the RM scores them, PPO updates the LLM to maximize reward
• Understand the KL divergence penalty: without it, the LLM would exploit the reward model, producing gibberish with high scores
• Learn why RLHF is hard to replicate: reward hacking, distributional shift, and the cost of human annotation are real bottlenecks

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.