GGUF

GGUF — explained in more detail

GGUF was introduced in 2023 as the successor to the older GGML format, to fix its predecessor’s problems (no unified metadata schema, awkward versioning). A GGUF file carries everything required to load a model: weights in different quantisation levels (e.g. Q4_K_M, Q5_K_S, Q8_0), the tokenizer vocabulary, architecture parameters and the chat template. It is loaded with llama.cpp or tools built on top of it like Ollama, LM Studio or KoboldCpp.

Example / Practical context

Hugging Face hosts thousands of GGUF conversions of open models (Llama, Qwen, Mistral, DeepSeek). Rule of thumb on quantisation: Q4_K_M is the well-established sweet spot between size and quality — a 7B model lands at around 4–5 GB instead of 14 GB unquantised. Q8_0 delivers near-full quality at twice the size; Q2/Q3 are only for emergencies when RAM/VRAM is extremely tight.

How it differs from related terms

GGML was the predecessor format — deprecated today, but older models still run on it. Safetensors is the Hugging Face format for unquantised weights, primarily for fine-tuning and cloud inference. AWQ and GPTQ are alternative quantisation schemes, mostly targeting GPU inference with vLLM or TGI — GGUF is explicitly built for CPU and mixed CPU/GPU setups.

Entdecke mehr

Blog

AI Workflows by Keyword: How We Make Recurring Routines Enforceable

A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.

Glossar

Quantization

Quantization reduces the numeric precision of model weights — e.g. from 16-bit float to 4-bit integer — making language models small and fast enough for consumer hardware, with manageable quality loss.

Lexikon

Running LLMs locally — hardware, tools, models

How to run language models on your own hardware — VRAM requirements, tooling (Ollama, LM Studio, llama.cpp, vLLM) and which models fit which GPU.

Themenuebersicht

KI-KonzepteLokales LLM (Konzepte)