Term
vLLM
vLLM is an open-source high-throughput inference server for LLMs — known for PagedAttention, continuous batching, and an OpenAI-compatible API endpoint, widely used in production self-hosting setups.
vLLM — explained
vLLM is built for server workloads: many concurrent requests, high tokens per second, GPU utilization as close to 100 % as possible. The core trick is PagedAttention — the KV cache is split into fixed blocks and managed like memory pages in an OS, which avoids fragmentation and makes continuous batching practical: running requests are merged token by token with newly incoming ones, instead of waiting for fixed batch boundaries. vLLM was originally developed at UC Berkeley.
Example / Practical context
On an A100 or H100, vLLM can run Llama 3.1 70B (via tensor parallelism across multiple GPUs) or Qwen 2.5 32B as a self-hosted backend for an internal ChatGPT-style assistant. The OpenAI-compatible endpoint (/v1/chat/completions) means existing clients (LangChain, LlamaIndex, custom SDKs) can switch over without code changes. Quantized models (AWQ, GPTQ) are supported out of the box.
Distinction from related terms
Ollama and LM Studio target single users and easy setup; vLLM targets servers with many concurrent users and maximum throughput. Hugging Face TGI and NVIDIA TensorRT-LLM compete in the same league as vLLM, with different emphases on hardware support, batching strategy, and license. llama.cpp, by contrast, is an inference engine on the CPU/edge level — not a server stack.
Entdecke mehr
Saving Tokens with Claude: 6 Principles That Make Experts Twice as Fast
How I turned my CLAUDE.md from a style guide into a token budget — 6 principles for lower cost, less waiting, and more honest reporting.
GlossarGPT4All
GPT4All is an open-source desktop application by Nomic AI for running large language models locally and offline on your own computer. It works without a GPU and without a cloud connection, so data never leaves the device.
LexikonRunning LLMs locally — hardware, tools, models
How to run language models on your own hardware — VRAM requirements, tooling (Ollama, LM Studio, llama.cpp, vLLM) and which models fit which GPU.