VRAM

VRAM — explained

Language models run fastest when all weights and the KV cache (the buffer for previous tokens) live in the GPU’s VRAM. Rough rule per model size at Q4 quantization: 7B ≈ 5 GB, 13B ≈ 9 GB, 34B ≈ 22 GB, 70B ≈ 40 GB — plus 1–4 GB for the KV cache depending on context length. If VRAM falls short, tools like llama.cpp offload layers into ordinary RAM — inference then becomes 2× to 20× slower depending on the offloading ratio.

Example / Practical context

An RTX 4090 with 24 GB VRAM comfortably runs 13B models with long context, and 34B Q4 with little headroom. A MacBook Pro M3 with 36 GB unified memory behaves similarly, because GPU and CPU share the same memory pool. Multiple GPUs can be combined through tensor or pipeline parallelism — vLLM and TGI handle this transparently. Cloud GPUs like A100 80 GB or H100 80 GB carry 70B models without issue.

Distinction from related terms

VRAM is not the same as system RAM — the latter is on a slower bus but plays the same role during offloading. Apple Silicon and integrated GPUs use unified memory: a single pool for CPU and GPU, which effectively raises the available “VRAM” for local LLMs. The bandwidth gap (GDDR6X ≈ 1 TB/s, DDR5 ≈ 90 GB/s) ultimately determines token throughput.

Entdecke mehr

Blog

AI Workflows by Keyword: How We Make Recurring Routines Enforceable

A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.

Glossar

GGUF

GGUF (GPT-Generated Unified Format) is the standard file format for quantised LLMs on the llama.cpp engine — a single `.gguf` file holds weights, tokenizer and metadata together.

Lexikon

Running LLMs locally — hardware, tools, models

How to run language models on your own hardware — VRAM requirements, tooling (Ollama, LM Studio, llama.cpp, vLLM) and which models fit which GPU.

Themenuebersicht

KI-KonzepteLokales LLM (Konzepte)