Term
Streaming (LLM)
Streaming means transmitting an LLM's response token by token in real time — the user sees the text word by word rather than only after the full generation completes.
Streaming — explained in more detail
Language models generate text autoregressively: token by token, one after another. Without streaming, the client waits for the entire response before receiving it in one chunk. With streaming, the API server pushes each newly generated token to the client immediately — typically via server-sent events (SSE) or WebSockets. This fundamentally changes the user experience: perceived waiting collapses to the time-to-first-token (often under a second), and the user can read long answers while they are still being produced. Programmatically, streaming requires a bit more code than a plain request/response call — but libraries such as the Anthropic, OpenAI, or Vercel AI SDKs wrap it cleanly.
Example / Practical context
ChatGPT, Claude.ai, Cursor chat — all well-known AI chats rely on streaming because a three-second “nothing happens” phase destroys the UX. In backend pipelines, by contrast, streaming is often unnecessary: a pipeline call usually wants the finished response, not reactions between tokens. In agent loops with tool use, streaming is also less critical because the loop must wait for structured tool calls anyway.
Distinction from related concepts
Batch inference is the opposite of streaming — deliberately no real time, but high throughput and low cost instead. Time-to-first-token (TTFT) is the key latency metric in a streaming context. Tokens-per-second then measures how fast generation continues. Both numbers vary widely between models and feed directly into API selection decisions.
Entdecke mehr
Headless without an API bill — how do you reach the best AI models for automation in 2026?
Provider comparison mid-2026: who has a headless mode, whose subscription still covers it — and why BYOK is the most stable foundation.
GlossarEU AI Act (KI-Verordnung)
Regulation (EU) 2024/1689 is the EU's first comprehensive law on artificial intelligence. It regulates AI systems by risk in four tiers, from prohibited practices to minimal risk, and sets obligations for providers and deployers.
LexikonFunction Calling / Tool Use
How an LLM uses tools: define a tool as a schema, the model picks the function and arguments, the result returns to the chat — the basis of every agent.