Running LLMs locally — hardware, tools, models
How to run language models on your own hardware — VRAM requirements, tooling (Ollama, LM Studio, llama.cpp, vLLM) and which models fit which GPU.
Redaktion
Lokales LLM
in KI-Werkzeuge
Tools zum Betreiben von Sprachmodellen auf eigener Hardware.
Glossary
GPT4All Lokales LLM
GPT4All is an open-source desktop application by Nomic AI for running large language models locally and offline on your own computer. It works without a GPU and without a cloud connection, so data never leaves the device.
Jan Lokales LLM
Jan is an open-source desktop application that lets you run large language models locally and fully offline on your own computer. It uses the llama.cpp engine and is positioned as a privacy-friendly ChatGPT alternative.
llama.cpp Lokales LLM
llama.cpp is an open-source C++ library for efficient LLM inference on CPU and GPU — the engine many local tools like Ollama and LM Studio are built on.
Llamafile Lokales LLM
Llamafile is a Mozilla project that packages a local language model as a single executable file — model weights and inference engine in one. That file runs on several operating systems without installation.
LM Studio Lokales LLM
LM Studio is a desktop application for running LLMs locally with a graphical interface — including a model browser, chat interface and OpenAI-compatible server.
Ollama Lokales LLM
Ollama is a tool for running language models locally — Llama, Mistral or Qwen — operated via a CLI, with an OpenAI-compatible HTTP server for applications.
Open WebUI Lokales LLM
Open WebUI is an open-source chat interface for local and remote LLMs — a ChatGPT-like browser UX that runs as a Docker container and talks to Ollama, OpenAI-compatible APIs and more.
TensorRT-LLM Lokales LLM
TensorRT-LLM is an open-source library from NVIDIA that optimizes LLM inference on NVIDIA GPUs. It compiles models into highly optimized runtime engines and offers in-flight batching, a paged KV cache and quantization for high throughput at low latency.
Text-Generation-WebUI Lokales LLM
Text-Generation-WebUI (oobabooga) is an open-source web interface for running LLMs locally. It bundles several inference backends, chat and completion modes and an OpenAI-compatible API under one common UI.
TGI (Text Generation Inference) Lokales LLM
TGI (Text Generation Inference) is an open-source toolkit from Hugging Face for serving large language models efficiently. It provides a production-ready inference server with continuous batching, token streaming and optimised attention methods.
vLLM Lokales LLM
vLLM is an open-source high-throughput inference server for LLMs — known for PagedAttention, continuous batching, and an OpenAI-compatible API endpoint, widely used in production self-hosting setups.
whisper.cpp Lokales LLM
whisper.cpp is an open-source C++ port of OpenAI's Whisper speech recognition — a compact native binary for local transcription on CPU, CUDA, Metal or Vulkan, with no Python dependency.