Term
TensorRT-LLM
TensorRT-LLM is an open-source library from NVIDIA that optimizes LLM inference on NVIDIA GPUs. It compiles models into highly optimized runtime engines and offers in-flight batching, a paged KV cache and quantization for high throughput at low latency.
TensorRT-LLM — explained in detail
TensorRT-LLM is an open-source library from NVIDIA for high-performance inference of large language models on NVIDIA GPUs. Rather than being a plain serving tool, it is primarily a compiler stack: a model (e.g. Llama, Mistral, Qwen) is translated via the Python API into an optimized TensorRT engine tailored to the target GPU, the desired precision and the parallelization scheme. That engine then runs through the C++ runtime for maximum speed.
Core optimizations include custom attention kernels, in-flight batching (new requests are added to the running batch at each generation step instead of waiting for previous requests to finish), a paged KV cache for efficient memory management, quantization (FP8, FP4, INT4 AWQ, INT8 SmoothQuant and more) and speculative decoding. For production there is a dedicated backend in the NVIDIA Triton Inference Server that bundles the TensorRT-LLM runtime with in-flight batching and paged KV caching.
Example / Practical use
A team that wants to serve a quantized 70B model to many concurrent users at minimal latency on H100 or A100 GPUs builds an FP8 engine with TensorRT-LLM and serves it via Triton. The extra effort compared to generic servers lies in the build step: the engine has to be generated per model, precision and GPU configuration, and is not portable across GPU generations. In return, throughput on NVIDIA hardware is best in class.
Distinction from related terms
vLLM pursues a similar goal (high throughput, paged KV cache via PagedAttention) but is broader in hardware compatibility and easier to run without a separate compile step — TensorRT-LLM is more deeply optimized for NVIDIA GPUs. Ollama and llama.cpp target simple local use, including CPU and Apple Silicon, rather than maximum server throughput. Topics like quantization and the KV cache are building blocks that TensorRT-LLM uses internally.
Entdecke mehr
Saving Tokens with Claude: 6 Principles That Make Experts Twice as Fast
How I turned my CLAUDE.md from a style guide into a token budget — 6 principles for lower cost, less waiting, and more honest reporting.
GlossarGPT4All
GPT4All is an open-source desktop application by Nomic AI for running large language models locally and offline on your own computer. It works without a GPU and without a cloud connection, so data never leaves the device.
LexikonRunning LLMs locally — hardware, tools, models
How to run language models on your own hardware — VRAM requirements, tooling (Ollama, LM Studio, llama.cpp, vLLM) and which models fit which GPU.