Running LLMs locally — hardware, tools, models
Why run LLMs locally — and who benefits
Cloud APIs are convenient, but they come with three weak spots that turn into showstoppers for many setups: data leaves your premises, every request costs money, and without internet you are stuck. If you regularly handle confidential client documents, run a coding assistant with tens of thousands of requests per month, or simply want to stay productive offline, local inference is the only real answer.
Getting started in 2026 is far smoother than it was two years ago. A mid-range GPU with 16 GB of VRAM can be added to a fairly modern machine for under €1,000, and open-weight models have caught up with the proprietary frontier on many tasks — Qwen 3.5 27B scores 72.4% on SWE-Bench Verified according to the Onyx leaderboard and fits on a single 16 GB GPU. What you can and cannot run locally comes down to three numbers: VRAM, quantization, model size. Understanding those three is half the battle.
The basics: VRAM, quantization, model size
Language models consist of billions of parameters (weights). To run inference, all those parameters need to sit in fast GPU memory (VRAM). If they do not fit, the system spills to system RAM or even SSD — and inference slows from acceptable to unusable. Honest VRAM math beats every marketing claim.
Quantization — the biggest lever
Models are originally trained in FP16 (16 bits per parameter, 2 bytes). For inference you can convert them to lower precision — that is quantization. Common levels:
- Q8 (8 bits, 1 byte/param) — minimal quality loss, half the memory of FP16
- Q5_K_M (~0.7 byte/param) — light, barely noticeable quality drop
- Q4_K_M (~0.55 byte/param) — the default for local setups, best size/quality trade-off
- Q3 / Q2 — only under heavy memory pressure; quality drops visibly
Rule of thumb: VRAM (GB) ≈ parameters (B) × bytes/param × 1.2 (the 20% headroom covers KV cache and activations). So an 8B model in Q4_K_M needs about 8 × 0.55 × 1.2 ≈ 5.3 GB. A 70B model in Q4_K_M lands around 40 GB — that fits on two 24 GB cards or a professional A100/H100, not on a single consumer GPU.
Context length eats more VRAM
Often overlooked: a filled context window also occupies memory. For an 8B model, 32k tokens of context can easily cost an extra 1–2 GB. If you work with large documents, budget VRAM for the KV cache — otherwise that shiny 128k context window only exists on paper.
VRAM table: what fits on which GPU
Realistic recommendations using Q4_K_M (default) and Q5_K_M (slightly higher quality):
| VRAM | Q4_K_M fits | Q5_K_M fits | Use case | |---|---|---|---| | 8 GB | 7B–8B (Llama 3.1 8B, Qwen 3 8B, Mistral 7B) | 7B tight | Chat, classification, coding hints | | 12 GB | 8B comfortably, 13B possible | 8B | Chat with bigger context, simple agents | | 16 GB | up to 27B (Gemma, Qwen) | up to 14B | Coding models, RAG with larger context | | 24 GB | up to 32B–34B | up to 22B | Demanding tasks, long contexts | | 48 GB | up to 70B | up to 48B | 70B-class as daily driver | | 2× 24 GB | 70B | 70B tight | Tensor parallelism (vLLM) |
Treat the values as orientation; exact requirements depend on architecture details (attention heads, KV heads with GQA) and context length. For model-precise math, see tools like the apxml VRAM calculator.
MoE models: fewer active parameters, full memory cost
A pitfall: mixture-of-experts models like Qwen3.5 (122B total / 10B active) or Mistral Large 3 (675B total / 41B active) need full memory for all parameters, even though only a fraction is active per token. They are fast but not small. For consumer GPUs, dense models in the 7B–34B range are almost always the better fit.
The tooling landscape: what runs where
Four tools dominate the field. They overlap, but solve different problems.
Ollama — the developer default
Ollama is by far the most widely used local LLM tool. One-line install, one command per model, automatic GPU offloading, an OpenAI-compatible HTTP API — that covers most workflows. It runs as a background service and powers coding assistants, RAG pipelines, and homegrown tools. Under the hood it builds on llama.cpp.
Weakness: under concurrent requests, Ollama does not scale — requests are processed sequentially. That is fine for single users or small teams, not for a production multi-user instance.
LM Studio — the GUI for non-developers
LM Studio is a desktop app with a model browser, chat interface, and built-in API server. If you want to demo models to colleagues or compare different ones, LM Studio gets you there without a terminal. It is also a friendly entry point for anyone who just wants to see what local LLMs can do.
Weakness: closed source and a bit less flexible for tuning than llama.cpp directly.
llama.cpp — the engine
llama.cpp is Georgi Gerganov’s C++ inference library, the foundation that almost every other local LLM runner is built on top of. If you want to squeeze maximum tokens per second, run GGUF models straight from the Hugging Face hub, or build custom server setups, you work with llama.cpp directly. Steeper learning curve, maximum control.
vLLM — the throughput server
vLLM is the right pick when multiple users hit the same model concurrently. With PagedAttention and continuous batching it reaches throughput numbers far above Ollama under load — benchmarks show up to 16× more tokens/s with five or more concurrent users. The price: vLLM needs NVIDIA GPUs (CUDA), is significantly more complex to set up, and for a single user the advantage is negligible.
Open WebUI — the chat frontend
Open WebUI is not an inference tool but a ChatGPT-style web interface that hooks into Ollama or any OpenAI-compatible endpoint. Multi-user login, prompt library, file uploads, RAG. Useful when an entire family or team should share a single local instance.
| Tool | Audience | Strength | Weakness | |---|---|---|---| | Ollama | Developers, single user | simple, OpenAI API | sequential requests | | LM Studio | Beginners, non-devs | GUI, model browser | closed source | | llama.cpp | Power users | maximum control, fast | learning curve | | vLLM | Multi-user production | throughput, batching | NVIDIA-only, complex | | Open WebUI | Frontend need | nice chat UI, RAG | UI only, needs backend |
Formats: GGUF vs. Safetensors
Two formats dominate — and they target different use cases.
GGUF (GPT-Generated Unified Format) is the format of the llama.cpp world. One file per model, containing weights, tokenizer, and metadata. Quantization levels live inside the format (Q4_K_M, Q5_K_M, Q8_0). If you use Ollama, LM Studio, or llama.cpp, you load GGUF.
Safetensors is the standard format of the Hugging Face world — safe (no pickle risk), fast to load, FP16/BF16 or pre-quantized via AWQ/GPTQ. vLLM, Transformers, and most training pipelines work with it. For pure inference on consumer hardware, GGUF is usually the more relaxed path; for production multi-user setups, Safetensors.
Model recommendations by VRAM tier (May 2026)
Concrete picks we recommend as daily drivers per GPU class — all in Q4_K_M unless noted otherwise.
8 GB VRAM
Llama 3.1 8B or Qwen 3 8B for general chat and classification. Mistral 7B as a compact alternative. For German workloads, look at SauerkrautLM variants that have been further trained on German data.
16 GB VRAM
This is where it gets interesting: Qwen 3.5 27B in Q4_K_M just fits and delivers exceptional coding results for its size. Alternatively, Gemma 4 (efficient, fast) or DeepSeek-Coder 16B for pure programming work.
24 GB VRAM
The sweet spot. Qwen 3.5 32B or DeepSeek V3.2-Exp (the smaller variant) live here. Longer contexts (16k–32k) run without acrobatics. For reasoning tasks, look at DeepSeek R1 or the Qwen reasoning variants.
48 GB+ VRAM
70B class as a daily driver. Llama 3.3 70B or the corresponding Llama 4 variant in Q4_K_M (~40 GB) leaves room for solid context. Two 24 GB cards with vLLM and tensor parallelism work just as well — and cost less than a single A6000.
Pitfalls
A few things that bite almost every local setup at some point.
Quantization losses are not linear
Q5 to Q4 often feels nearly identical. Q4 to Q3, by contrast, becomes visible — hallucinations rise, logic tasks tip over. Q2 is not recommended outside experiments. For important tasks, prefer a smaller model at higher precision over a large one with aggressive quantization.
CPU inference is slow but usable
Without a GPU you can run llama.cpp in CPU mode. An 8B model on a modern CPU runs at 5–10 tokens/s — fine for asynchronous use, painful for streaming chat. Apple Silicon Macs with unified memory are the exception: an M3 Max with 64 GB outpaces every 16 GB consumer GPU on large models, because RAM and VRAM share the same physical address space.
Top-tier cards run hotter than you think
A 24 GB card under full load draws 350–450 W with corresponding noise and heat. If your workstation sits in the living room, measure first — or set up a dedicated inference machine on an old server motherboard.
The OpenAI API is not quite the OpenAI API
Ollama, LM Studio, and vLLM all speak “OpenAI-compatible” — but each only a subset. Structured output, function calling, streaming details, and tool-use semantics differ. When porting cloud code to local, walk every endpoint and verify behavior point by point.
Model updates can break behavior
An Ollama upgrade or a fresh model build can change behavior (different tokenizer, different default templates). For production setups, pin model versions and set templates explicitly — never rely on defaults.
Three typical setups
Setup A: Solo developer with coding focus
Hardware: RTX 4080 or 4090 (16/24 GB). Tooling: Ollama plus Open WebUI as chat surface. Models: Qwen 3.5 27B or 32B in Q4_K_M for coding, plus a small 7B model for fast classification tasks. IDE integration via Ollama’s OpenAI API. Cost: roughly €1,500–€2,500 one-time for the GPU.
Setup B: Privacy-strict law firm or medical practice
Hardware: workstation with 48 GB VRAM (one A6000 or two 4090s with tensor parallelism). Tooling: vLLM as server, Open WebUI as frontend with multi-user login. Model: Llama 3.3 70B or a German SauerkrautLM 70B variant in Q4_K_M. Data never leaves the building, all client documents stay local. Cost: €5,000–€10,000 one-time instead of recurring cloud fees.
Setup C: Family server for offline access
Hardware: a used mini-PC or Mac mini with 32 GB unified memory. Tooling: LM Studio in server mode or Ollama, Open WebUI for the family. Model: Llama 3.1 8B or Gemma 4 — enough for homework help, translations, recipes. No client confidentiality involved, but also no US servers reading along.
Conclusion
Local LLMs in 2026 are a real tool, no longer a hobbyist project. With 8–16 GB of VRAM, Ollama plus a 7B/27B model gives you a system that replaces most cloud requests — at zero token cost and full data control. For multi-user scenarios, the path leads through vLLM and 48+ GB VRAM, but then with a real throughput advantage.
The most important rule: prefer a smaller model at higher precision over an oversized one with aggressive quantization. And when buying hardware, do not forget the KV cache — the nominally fitting GPU is often not enough once 32k of context is filled. Size carefully and a local setup will carry you for years.
Entdecke mehr
Why AI Models Find Different Code Problems
Three frontier models, the same 1000-line script, three different finding lists — and why that very spread makes multi-orchestration strong.
GlossarGGUF
GGUF (GPT-Generated Unified Format) is the standard file format for quantised LLMs on the llama.cpp engine — a single `.gguf` file holds weights, tokenizer and metadata together.
LexikonFunction Calling / Tool Use
How an LLM uses tools: define a tool as a schema, the model picks the function and arguments, the result returns to the chat — the basis of every agent.