Lokales LLM (Konzepte)
Konzepte rund um lokalen Modellbetrieb — Formate, Quantisierung, Hardware.
- Batch Inference Lokales LLM
Batch inference is the bundled processing of many prompts in one pass — locally on a GPU or as an asynchronous API job — trading real-time latency for throughput and cost.
- GGUF Lokales LLM
GGUF (GPT-Generated Unified Format) is the standard file format for quantised LLMs on the llama.cpp engine — a single `.gguf` file holds weights, tokenizer and metadata together.
- Quantization Lokales LLM
Quantization reduces the numeric precision of model weights — e.g. from 16-bit float to 4-bit integer — making language models small and fast enough for consumer hardware, with manageable quality loss.
- VRAM Lokales LLM
VRAM (Video RAM) is the dedicated memory on a GPU — the single most important hardware figure for running language models locally, because ideally the whole model and its context must fit inside it.