KV Cache

KV Cache — explained in detail

The KV cache (key-value cache) is an optimisation mechanism used during the inference of Transformer language models. In the attention computation, each token is turned into three vectors: query, key and value. To predict a new token, the model compares the query of the current token with the keys of all previous tokens and uses this to weight their values.

Without intermediate storage, the model would have to recompute the key and value vectors for the entire preceding sequence with every newly generated token. Because language models produce text token by token (autoregressively), this effort would grow quadratically with each position. The KV cache stores the key and value vectors once computed and makes them available again at every subsequent step. For each new token, then, only its own vectors need to be calculated.

The trade-off is memory: the cache grows linearly with sequence length and model size and can become the limiting factor for output token throughput with long contexts. Techniques such as multi-query and grouped-query attention reduce this demand by letting several attention heads share common key and value vectors.

Example / Practical use

For a chatbot generating a long answer, the KV cache ensures that each additional word appears quickly and with constant compute per token. If the cache were cleared, the model would have to reprocess the entire prior history. This is exactly why a long conversation or a large context window noticeably occupies memory on the hardware: the KV cache for all tokens must be held simultaneously.

Distinction from related terms

The KV cache should not be equated with the context window. The context window describes the maximum number of tokens a model can process; the KV cache is the concrete technical data structure that holds the intermediate results for those tokens during generation. It also differs from application-level caching (such as prompt caching), which reuses whole requests or partial prompts — the KV cache instead operates within a single generation run at the level of the attention layers.

Entdecke mehr

Blog

AI Workflows by Keyword: How We Make Recurring Routines Enforceable

A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.

Glossar

Reasoning Effort

Reasoning Effort is a control parameter on modern reasoning models that sets how much internal step-by-step thinking (thinking tokens) a model spends before answering — higher levels raise quality but also latency and cost.

Lexikon

LLM Hallucinations — Causes and Remedies

Why LLMs confidently invent falsehoods, what types of hallucinations exist, and which remedies actually help — RAG, source enforcement, verification.

Themenuebersicht

KI-KonzepteLLM-Grundlagen