QLoRA

QLoRA — in more detail

QLoRA (Quantized Low-Rank Adaptation) was introduced in 2023 by researchers at the University of Washington and combines two ideas: parameter-efficient LoRA adapters with aggressive 4-bit quantization of the frozen base model. It adds technical tricks such as the NF4 data type (Normal Float 4), double quantization and paged optimizers. Together they slash GPU memory needs — a 65-billion-parameter model that normally requires multiple high-end GPUs can be fine-tuned on a single 48 GB GPU.

Example / practical context

QLoRA put open-source fine-tuning within reach of many practitioners. A 13B model like Llama 3 or Mistral can be adapted to a specific domain on consumer hardware (24 GB VRAM) in a few hours with QLoRA. Typical stack: bitsandbytes for the quantization plus Hugging Face’s peft and transformers for training. The resulting adapter file stays small and can be loaded onto a non-quantized base model at inference time.

Delineation from similar terms

LoRA operates on an unquantized model — same adapter mechanism, higher memory footprint. Classic post-training quantization (PTQ) such as GPTQ or AWQ shrinks an already-trained model for faster inference, without training. QLoRA is the bridge: it trains on top of a quantized base model without giving up quality.

Entdecke mehr

Blog

AI Workflows by Keyword: How We Make Recurring Routines Enforceable

A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.

Glossar

DPO (Direct Preference Optimization)

DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.

Lexikon

Fine-Tuning Explained — When, How, With What

When fine-tuning pays off, which methods exist (SFT, DPO, LoRA, QLoRA), and what AMD vs. NVIDIA hardware actually means in practice.

Themenuebersicht

KI-KonzepteTraining & Fine-Tuning