Main area
Agency
ki-konzepte

Tokens and Tokenization in LLMs Explained

Redaktion ·

Tokens and Tokenization in LLMs Explained

A large language model (LLM) doesn’t read text the way you do. It sees tokens — small text units into which every prompt and every response is split. Tokens are the currency LLMs compute in: they determine what you pay, how much fits into a context window, and how fast a response is produced. Anyone working with LLMs — whether via the API or while prompting — should understand what a token is and how the splitting works.

What a token is

A token is a text unit the model treats as its smallest processing unit. Important: a token is not the same as a word. It’s often a word fragment, sometimes a whole word, sometimes just a punctuation mark or a space.

A few examples make this tangible. The English word “token” is a single token. “tokenization”, by contrast, is often split into several pieces — for example “token” + “ization”. Common words get their own token; rare or compound words are cut into known parts. The detailed term is in the glossary under token.

How a tokenizer splits text

The splitting is done by the tokenizer. Most modern LLMs use a method called Byte-Pair Encoding (BPE) or a variant of it.

The idea is simple: BPE starts with individual characters and iteratively merges the most frequent adjacent pairs into new units. Across huge amounts of text this builds a vocabulary in which common words and word parts each become a token, while rare sequences consist of several tokens. That’s a good compromise between “every character on its own” (too many tokens) and “every word as a token” (vocabulary too large, unknown words unrepresentable).

In practice this means: you can usually try out a provider’s tokenizer online and see how your specific text is split. That’s the most reliable way to determine a token count — more important than any rule of thumb.

Rules of thumb for estimating tokens

For a rough estimate, two values have become established that hold for English text:

  • 1 token ≈ 4 characters
  • 1 token ≈ 0.75 words (inversely: 100 words ≈ 133 tokens)

These numbers are approximations, not an exact conversion. They help gauge an order of magnitude — for example whether a document roughly fits into a context window.

Why tokens drive cost

LLM APIs bill per token, not per word or character. There’s usually a price per 1,000 or per 1 million tokens. The split into two directions is decisive:

  • Input tokens — everything you send into the model: system prompt, instruction, context, prior conversation.
  • Output tokens — everything the model generates.

Output tokens are significantly more expensive than input tokens with most providers. That has direct consequences for prompting: a long context (lots of input) is often cheaper than a long generated answer (lots of output). To cut costs, first trim the forced output length, not necessarily the input.

Tokens, context window, and speed

Tokens determine two more hard limits:

Context window. Every model has a maximum context window, measured in tokens. Input and prior history plus the planned output must fit into it together. If the history is too long, it has to be trimmed or summarized — otherwise the request fails.

Latency. A model generates output token by token. More output tokens directly mean more compute time and thus a slower response. A short answer arrives faster than a long essay — regardless of content.

Together this explains why token awareness isn’t just a cost question, but also decides feasibility (does it fit in the window?) and user experience (how fast does it respond?).

FAQ

Is a token the same as a word? No. A token is often a word fragment. Common short words are one token; longer or rare words are split into several tokens. As a rough rule for English: one token equals about 0.75 words or roughly four characters.

How do I count tokens reliably? Use the tokenizer of the specific model — many providers offer it online or via API. That’s more accurate than any rule of thumb, because every model has its own vocabulary. Don’t rely on the word count.

Why does German text need more tokens? Mainly because of long compound words without spaces, which the tokenizer has to cut into several pieces. Umlauts and English-optimized vocabularies also raise the token count per word compared to English.

What costs more — input or output? Output tokens are more expensive than input tokens with most providers. A long context is therefore often cheaper than a long generated answer. To cut costs, limit the output length first.

What does the context window have to do with tokens? The context window is the maximum token count that input, prior history, and planned output together must not exceed. If it gets too tight, the history has to be trimmed or summarized.

Themenuebersicht

KI-KonzepteLLM-Grundlagen