Term
Sampling (LLM)
Sampling is the weighted drawing of the next token from a language model's probability distribution — governed by temperature, top-p and top-k. It produces the variability between two runs.
Sampling (LLM) — explained in detail
A language model produces its output token by token. At each step the model does not compute a single fixed token but a probability distribution over all possible next tokens in its vocabulary — every candidate is assigned a probability. Sampling is the act of actually picking one token from that distribution: the model draws a candidate in a weighted way — randomly, but weighted by probability. Tokens with a higher probability are drawn more often, yet rarer ones can still appear.
How widely this selection spreads is controlled by three parameters working together:
- Temperature scales the distribution before drawing. Low values sharpen it (the most likely tokens dominate), high values flatten it (even unlikely tokens get a real chance).
- Top-k hard-limits the choice to the k most likely candidates and discards the rest.
- Top-p (nucleus sampling) dynamically takes as many tokens as needed until their cumulative probability reaches the threshold p.
Only from this reshaped remaining set is the token drawn. Sampling is therefore the lever that separates language models from purely deterministic prediction.
Example / Practical use
Sampling is why the same model can return different outputs across two runs with an identical prompt. Ask a model to analyse the same code twice and it may flag one vulnerability the first time and a different one the second — not because it “knows more,” but because a different token was drawn at an early fork. This path dependence propagates: a single divergently sampled token early in the answer steers the entire remaining text in another direction.
In practice this is used deliberately: anyone needing reproducible, concise answers (e.g. structured data extraction) lowers temperature and top-p to minimise spread. Anyone wanting several solution variants or creative suggestions raises them. Repeatedly sampling the same question and then comparing the answers is also a simple way to detect uncertain or unstable model statements.
Distinction from related terms
Greedy decoding is the counterpart to sampling: instead of drawing in a weighted way, it always picks the single most likely token at each step. That makes the output deterministic and reproducible, but often repetitive and uncreative — and it does not necessarily find the overall best sequence.
Temperature, top-p and top-k are not alternatives to sampling but its control parameters: they shape the distribution that sampling draws from. Sampling is the overarching mechanism, these parameters are the levers attached to it. Greedy decoding can be seen as a special case — sampling at temperature 0.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarReasoning Effort
Reasoning Effort is a control parameter on modern reasoning models that sets how much internal step-by-step thinking (thinking tokens) a model spends before answering — higher levels raise quality but also latency and cost.
LexikonLLM Hallucinations — Causes and Remedies
Why LLMs confidently invent falsehoods, what types of hallucinations exist, and which remedies actually help — RAG, source enforcement, verification.