Training & Fine-Tuning

DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.

Continued training of a pretrained model on a smaller, specialized dataset to adapt style, format or domain knowledge in a targeted way.

Low-Rank Adaptation — parameter-efficient fine-tuning that trains only small additional matrices instead of all model weights.

PEFT bundles methods that adapt a pretrained LLM by training only a small fraction of its weights — typically under 1%, instead of all billions of parameters.

Extension of LoRA that quantizes the base model to 4 bits — allowing even very large LLMs to be fine-tuned on a single GPU.

Reinforcement Learning from Human Feedback — training procedure that aligns models with helpful and safe behavior using human preference comparisons.

SFT is the supervised post-training of a pretrained language model on a dataset of input-output pairs — the step that turns a base model into a usable assistant.

Synthetic data is artificially generated training or evaluation data — typically produced by a language model itself to extend datasets without needing real-world sources.

An adapter is a small, freshly added network module injected into a frozen language model — only these modules are trained, while the base model itself stays unchanged.

Catastrophic forgetting describes the effect that a neural network loses previously acquired knowledge — wholly or partially — when it is trained further on new data; a key problem when fine-tuning language models.

Continued pretraining is the further training of an already pretrained language model on large amounts of domain text — before any classic fine-tuning begins.

Instruction tuning is a variant of supervised fine-tuning in which a language model is trained on a broad mix of instruction-response pairs — so it reliably follows instructions across arbitrary tasks.

Knowledge distillation is a training technique in which a small student model learns from the behaviour of a large teacher model — aiming for comparable quality at significantly lower resource cost.

ORPO is a training method that unifies supervised fine-tuning and preference optimisation in a single step — leaner than the classic SFT-plus-RLHF pipeline.

Prefix tuning is a PEFT method that trains a sequence of learned "virtual tokens" prepended to the input rather than any model weights — the base model itself stays frozen.