Term
Batch Inference
Batch inference is the bundled processing of many prompts in one pass — locally on a GPU or as an asynchronous API job — trading real-time latency for throughput and cost.
Batch inference — explained in more detail
When a language model processes only a single prompt at a time, the GPU sits idle most of the time — memory and compute units are underused. Batch inference therefore groups multiple requests so they flow through the model in parallel. The marginal cost of each additional prompt is small because the weights are loaded into memory anyway. Providers such as OpenAI and Anthropic offer dedicated batch APIs: requests are submitted as a file and returned within hours, at a meaningful discount (often 50%) compared with the real-time tariff. Locally, this bundling is typically handled by a serving engine such as vLLM.
Example / Practical context
Classic case: 200,000 product descriptions in an e-commerce database need to be rewritten into shorter variants by an LLM. Real-time via the normal API would be costly and slow. Instead, all requests are written into a JSONL file and submitted to the batch API; the result lands by the next morning — at half the token rate. The same pattern applies to evaluation runs, re-indexing, or generating synthetic training data.
Distinction from related concepts
Real-time inference answers individual requests immediately, which is required for chats, agents, and IDE integrations — but costs more per token and cannot be parallelised arbitrarily. Streaming returns the answer token by token, so it is a variant of real-time inference. Continuous batching (vLLM, TGI) is the technical implementation of batch inference in a real-time serving setup: requests are dynamically grouped into batches without noticeably delaying any one response.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarGGUF
GGUF (GPT-Generated Unified Format) is the standard file format for quantised LLMs on the llama.cpp engine — a single `.gguf` file holds weights, tokenizer and metadata together.
LexikonRunning LLMs locally — hardware, tools, models
How to run language models on your own hardware — VRAM requirements, tooling (Ollama, LM Studio, llama.cpp, vLLM) and which models fit which GPU.