Term
Quantization
Quantization reduces the numeric precision of model weights — e.g. from 16-bit float to 4-bit integer — making language models small and fast enough for consumer hardware, with manageable quality loss.
Quantization — explained
A 70B model at full fp16 precision occupies roughly 140 GB of VRAM — not a typical consumer setup. With quantization, weights are converted into a more compact number format: 8-bit (Q8), 5-bit (Q5), 4-bit (Q4), down to 2-bit. The same 70B model fits into about 40 GB at Q4. Quality loss is usually unmeasurable at Q8, generally acceptable at Q4, and grows noticeable below that. Methods like GPTQ, AWQ, and the GGUF K-quants extract significantly more quality at low bit widths than naive rounding.
Example / Practical context
If you run Llama 3.1 8B or Qwen 2.5 14B locally in Ollama or LM Studio, you almost always load a quantized GGUF variant (Q4_K_M being the common default). A 14B Q4 model then runs on a 16 GB GPU or, with CPU+RAM offloading, even without a dedicated graphics card. On the inference-server side, vLLM, TGI, and TensorRT-LLM use AWQ or GPTQ for higher throughput.
Distinction from related terms
Quantization happens after training and does not change the model’s content — unlike fine-tuning, which actually re-learns the weights. Knowledge distillation, in contrast, trains a smaller model from scratch to imitate a larger one. Pruning removes whole weights instead of just representing them less precisely.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarGGUF
GGUF (GPT-Generated Unified Format) is the standard file format for quantised LLMs on the llama.cpp engine — a single `.gguf` file holds weights, tokenizer and metadata together.
LexikonRunning LLMs locally — hardware, tools, models
How to run language models on your own hardware — VRAM requirements, tooling (Ollama, LM Studio, llama.cpp, vLLM) and which models fit which GPU.