Term
ORPO (Odds Ratio Preference Optimization)
ORPO is a training method that unifies supervised fine-tuning and preference optimisation in a single step — leaner than the classic SFT-plus-RLHF pipeline.
ORPO — explained in more detail
The classic post-training pipeline for a chat model looks like this: SFT on question-answer pairs first, then a second phase with RLHF or DPO in which the model is further tuned on preference pairs (“answer A better than B”). That is expensive: two training phases and potentially a separate reward model. ORPO (Hong et al., 2024) packs both into one step. The loss combines classic cross-entropy on the preferred response with an odds-ratio term that additionally discourages the model from making the rejected response more likely. The method does not need a reference model — unlike DPO, which keeps the original model around for comparison — making it significantly more memory-efficient.
Example / Practical context
Typical use: an open-source base model should be turned into a chat assistant, with limited resources and a dataset of preference pairs (such as UltraFeedback or an in-house collection). Instead of two separate training runs (SFT, then DPO), one ORPO run suffices. Hugging Face’s trl library supports ORPO directly, often combined with LoRA for additional efficiency.
Distinction from related concepts
RLHF is the original method — reinforcement learning on a separately trained reward model: powerful, but complex. DPO (Direct Preference Optimization) eliminates the reward model by translating preferences directly into a loss term, but still needs a reference model. ORPO goes a step further and unifies SFT and preference optimisation. KTO and IPO are further related methods with different loss formulations.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarDPO (Direct Preference Optimization)
DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.
LexikonFine-Tuning Explained — When, How, With What
When fine-tuning pays off, which methods exist (SFT, DPO, LoRA, QLoRA), and what AMD vs. NVIDIA hardware actually means in practice.