Term
Tokenizer
A tokenizer is the program that splits text into tokens before a language model can process it — it determines how many tokens a piece of text costs.
Tokenizer — explained in detail
Before a language model can read a prompt, the text must be split into tokens — small units that usually correspond to word fragments. That is the tokenizer’s job. Each model family ships its own: OpenAI’s GPT models use tiktoken, Anthropic’s Claude models switched to a new tokenizer with Opus 4.7, Meta models use SentencePiece variants. Because different tokenizers split the same text into different numbers of tokens, the cost per request varies across models — even when the per-token price is identical.
Practical relevance
A concrete example: Opus 4.7 uses a new tokenizer that produces up to 35% more tokens from the same text compared to Opus 4.6’s tokenizer. The per-token price is identical ($5 / MTok input, $25 / MTok output), but the real cost per request rises by up to 35% because more tokens are generated. Code and structured data are hit hardest. Knowing the tokenizer characteristics of your model lets you shorten prompts, use caching more effectively, and estimate actual costs more accurately.
Distinction from related terms
A tokenizer is not the same as a token: the tokenizer is the tool, the token is the result. Also not to be confused with embeddings — those are created after tokenization, when the model converts tokens into numerical vectors.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarReasoning Effort
Reasoning Effort is a control parameter on modern reasoning models that sets how much internal step-by-step thinking (thinking tokens) a model spends before answering — higher levels raise quality but also latency and cost.
LexikonLLM Hallucinations — Causes and Remedies
Why LLMs confidently invent falsehoods, what types of hallucinations exist, and which remedies actually help — RAG, source enforcement, verification.