Term
Prefix Tuning
Prefix tuning is a PEFT method that trains a sequence of learned "virtual tokens" prepended to the input rather than any model weights — the base model itself stays frozen.
Prefix tuning — explained in more detail
Introduced in 2021 by Li & Liang (Stanford), prefix tuning uses an elegant idea: instead of learning weights or adapter modules, you learn a sequence of continuous vectors that act as a prefix to the actual input in every attention layer. These “prefix tokens” are not real tokens from the vocabulary — they are freely trainable embeddings in the model’s latent space. The training footprint is tiny (typically under 0.1% of the model’s parameters), and multiple prefixes can be kept in parallel — one per use case. A closely related variant is prompt tuning (Lester et al., 2021), where a learned vector prefix exists only at the input layer — even lighter, but less expressive.
Example / Practical context
Practical niche: when a model has to be adapted to many small, slightly different tasks and the memory footprint per task must be minimal. Classically used for multi-task setups in research and production systems with dozens of tenants. In practice, LoRA has largely displaced prefix tuning — LoRA is typically just as efficient, more stable to train, and mergeable into the base model.
Distinction from related concepts
LoRA injects low-rank matrices into the attention weights and is today’s de-facto standard. Classic adapters insert MLP modules between layers. Prefix tuning changes neither weights nor architecture — it only extends the context with learned vectors. Prompt tuning is the minimal variant of it — prefix tuning learns vectors at every layer, prompt tuning only at the input.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarDPO (Direct Preference Optimization)
DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.
LexikonFine-Tuning Explained — When, How, With What
When fine-tuning pays off, which methods exist (SFT, DPO, LoRA, QLoRA), and what AMD vs. NVIDIA hardware actually means in practice.