Term
LoRA
Low-Rank Adaptation — parameter-efficient fine-tuning that trains only small additional matrices instead of all model weights.
LoRA — in more detail
LoRA stands for Low-Rank Adaptation, a method introduced by Microsoft Research in 2021 to adapt large language models efficiently. Instead of updating every model weight, LoRA freezes the original and trains small low-rank matrices that are inserted in parallel into selected layers (typically the attention projections). This cuts training cost dramatically — often to under 1 % of the trainable parameters — while reaching quality comparable to full fine-tuning.
Example / practical context
A 7-billion-parameter model like Llama 3 can be fine-tuned with LoRA on a single consumer GPU (24 GB VRAM). Training is typically done with Hugging Face’s peft library; the output is a small adapter file (a few MB up to a few hundred MB) that is loaded on top of the base model at inference time. Multiple LoRA adapters for different tasks can share the same base model — at inference you simply select the matching adapter.
Delineation from similar terms
LoRA is a specific flavor of parameter-efficient fine-tuning (PEFT) and a relative of prefix-tuning or adapters. QLoRA extends LoRA with 4-bit quantization of the base model and reduces memory usage further. Full fine-tuning updates every weight, is slightly stronger in quality, but orders of magnitude more expensive.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarDPO (Direct Preference Optimization)
DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.
LexikonFine-Tuning Explained — When, How, With What
When fine-tuning pays off, which methods exist (SFT, DPO, LoRA, QLoRA), and what AMD vs. NVIDIA hardware actually means in practice.