Main area
Agency
ki-konzepte

Temperature, Top-p and Sampling in LLMs

Redaktion ·

A language model guesses at every step: which word comes next? But it doesn’t guess blindly. For each possible next token the model computes a probability — a distribution over the entire vocabulary. For the prompt „The sky is …” maybe 60% of the probability lands on „blue”, 8% on „cloudy”, and a tiny remainder is spread across thousands of other words. Which word is actually chosen in the end is decided by sampling — and this is exactly where temperature, top-p and top-k come in.

These parameters don’t change what the model knows. They only change how boldly it draws from its own probabilities. Understand them and you understand why the same model sometimes returns the exact same answer stubbornly and sometimes veers off creatively — and which setting you want for code, facts or creative writing.

From logits to probability

Before any sampling happens, the model outputs a raw value for each vocabulary token — the logits. These aren’t probabilities yet: they can be negative and don’t sum to anything meaningful. A function called softmax turns these raw values into a clean distribution that adds up to 100%. Only on this distribution do the sampling parameters operate.

Temperature: sharpen or flatten the distribution

Temperature is a scalar that divides the logits before the softmax. The effect:

  • Low temperature (toward 0). The distribution gets sharper. The high probabilities gain even more mass, the low ones shrink. The model becomes predictable and factual — it almost always grabs the most probable token. At temperature 0 the choice becomes practically deterministic.
  • Temperature 1. Pass-through. The model samples from its native distribution, unchanged.
  • High temperature (above 1). The distribution gets flatter. The top token loses its lead, less likely tokens suddenly become reachable. The model gets more varied, more creative — and riskier, because nonsense becomes more likely too.

The „temperature” image fits: cold = ordered and rigid, hot = mobile and chaotic.

Top-p (nucleus sampling): cut off by confidence

Top-p, also called nucleus sampling, takes a different approach. Instead of reshaping the distribution, it limits the choice. Top-p selects the smallest set of the most probable tokens whose cumulative probability reaches the threshold p — this „core” (nucleus). The draw then happens from this set; the long tail of unlikely tokens drops away entirely.

The clever part: the set is adaptive. If the model is very sure (one token has 95%), the nucleus consists of exactly that one. If it’s unsure (twenty tokens at 5% each), the nucleus spans many. Top-p adapts to the model’s confidence instead of imposing a fixed count.

Top-k: the fixed cap

Top-k is the simpler variant: it restricts the choice to the k most probable tokens, regardless of how the probabilities are spread. Top-k = 40 means only the 40 most probable tokens are eligible; the rest are ignored.

The weak spot is exactly this rigidity: k is fixed, independent of context. On an unambiguous question the 40 allowed tokens are too many (nonsensical options sneak in); on an open question maybe too few. That’s why top-p has prevailed in practice — it solves exactly this problem.

How the parameters interact

Temperature and top-p/top-k act on different levels and are often combined. The usual processing order: first temperature scales the distribution, then top-p (or top-k) trims the choice, then the draw happens.

In practice you rarely crank both hard at once. A common rule of thumb: for facts, code and Q&A a low temperature and a high top-p — this keeps the model on the most probable answers but leaves enough room for natural-sounding text. For creative writing a higher temperature, so the model dares surprising turns.

Note: some models restrict these parameters

The sampling parameters aren’t a universal standard. Some newer reasoning models no longer allow setting them freely. A concrete example: Anthropic’s current top models Claude Opus 4.7 and 4.8 have fully removed temperature, top_p and top_k — a request with these fields returns an error (as of June 2026). The idea behind it: for models that „think” internally, behavior should be steered via the task description and effort levels, not via a raw sampling setting. So anyone used to setting temperature everywhere should check the specific model’s docs before relying on it — the parameters explained here don’t necessarily apply to every model.

FAQ

What exactly does temperature do in an LLM? It scales the probability distribution over the possible next tokens before the draw. Low temperature makes the distribution sharper (deterministic, factual), high temperature makes it flatter (more varied, creative, riskier). At temperature 0 the model almost always picks the most probable token.

What’s the difference between top-p and top-k? Top-k restricts the choice to a fixed number k of the most probable tokens. Top-p (nucleus sampling) restricts it to the smallest set whose probabilities together reach the threshold p — and this set is adaptive: small when the model is confident, large when it’s unsure.

Which setting do I use for code or facts? A low temperature, often combined with a high top-p. This makes the model reliably grab the most probable, most correct tokens and not stray into nonsensical alternatives. For creative writing you turn the temperature up.

Should I change temperature and top-p at the same time? Usually not both hard at once. The common approach is to use one parameter as the main lever (often temperature) and leave the other at a moderate default. Setting both aggressively at once makes behavior hard to predict.

Why can’t I set temperature on some models? Some newer reasoning models have deliberately removed the sampling parameters — for example Claude Opus 4.7 and 4.8, which reject temperature, top_p and top_k (as of June 2026). Behavior there is steered via the task description and effort levels. Always check the specific model’s docs before relying on these parameters.

Themenuebersicht

KI-KonzepteLLM-Grundlagen