Term
DPO (Direct Preference Optimization)
DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.
DPO — explained in more detail
Direct Preference Optimization was introduced in 2023 by Rafailov et al. and has since become the pragmatic alternative to RLHF. Instead of first training a reward model and then optimizing the language model via PPO, DPO derives a closed-form loss directly from preference pairs: each prompt comes with a preferred and a rejected answer. The model learns to raise the probability of the preferred answer over the rejected one — relative to a frozen reference copy of itself.
Example / Practical use
In practice this means a single training run on a standard SFT pipeline (typically via Hugging Face TRL or comparable libraries), no reward model, no PPO tuning. DPO is markedly more stable and less resource-hungry than RLHF, and has become the default open-source alignment method in subsequent years — used for Llama 3, Mistral and many Hugging Face models.
How it differs from similar concepts
Unlike RLHF, DPO drops the reward model and the PPO step. Unlike SFT, DPO learns not only the correct answer but also explicitly the rejected one — relative to each other. Successors like ORPO (Odds Ratio Preference Optimization) try to fuse SFT and preference learning into a single step.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarFine-Tuning
Continued training of a pretrained model on a smaller, specialized dataset to adapt style, format or domain knowledge in a targeted way.
LexikonFine-Tuning Explained — When, How, With What
When fine-tuning pays off, which methods exist (SFT, DPO, LoRA, QLoRA), and what AMD vs. NVIDIA hardware actually means in practice.