Term
Prompt Caching
Prompt caching is an API feature in which a provider stores recurring prompt prefixes — making subsequent requests cheaper and faster because the cached portion is not reprocessed.
Prompt caching — explained in more detail
For long system prompts, tool definitions or attached documents, the same text block often repeats across thousands of requests. Without caching, it is tokenised and pushed through the first layers of the model every single time. Prompt caching stores the internal states after that block — so-called key-value cache entries — and reuses them on the next request. Anthropic, OpenAI and Google offer this as an API feature. The effect is dramatic: cache hits typically cost only 10% of the regular input token price and noticeably reduce time-to-first-token. Caches have a limited lifetime (usually 5 minutes passive, hours to days explicit) and must sit at the beginning of the prompt.
Example / Practical context
Classic case: a coding agent runs a multi-turn dialogue with 30k tokens of repository context. Without caching, every follow-up call costs the full 30k input tokens. With prompt caching, repository context and system prompt are cached after the first call — the next twenty turns only pay the full rate for the genuinely new tokens (user question + prior answers), while the rest counts at the cache-hit price.
Distinction from related concepts
Classic response caching stores complete answers for identical requests — but loses all value as soon as the input varies even slightly. Embedding caching stores precomputed vector representations and is a different cache layer entirely. Prompt caching does not match across identical requests but across identical prompt prefixes — making it far more practical in agent settings.
Entdecke mehr
Headless without an API bill — how do you reach the best AI models for automation in 2026?
Provider comparison mid-2026: who has a headless mode, whose subscription still covers it — and why BYOK is the most stable foundation.
Glossar$/MTok (Cost per Million Tokens)
Standard pricing unit for AI APIs — cost in US dollars per one million processed tokens. Listed separately for input, output and sometimes cache.
LexikonFunction Calling / Tool Use
How an LLM uses tools: define a tool as a schema, the model picks the function and arguments, the result returns to the chat — the basis of every agent.