Main area
Agency
arbeiten-mit-ki

Managing Your Token Budget — the Biggest Cost Levers for LLMs

Redaktion ·

If you work seriously with LLMs, the token budget is where playing around turns into an invoice. The basics — what a token is, how pricing works — are covered in other articles (see AI Pricing Explained). This one is about practice: which levers cut your costs the most without hurting quality? The answer is usually not “smaller model” but “fewer tokens, used more cleverly.”

Tokens are the billing unit

LLM APIs bill per token — per text fragment that goes in and comes out. Two numbers appear on every invoice:

  • Input tokens: everything you send to the model (system prompt, history, question, documents).
  • Output tokens: everything the model sends back.

The crucial point many overlook: output is usually much more expensive than input. On current top models the output price is often four to five times the input price. As a dated snapshot (Anthropic, as of 2026): an Opus model at around 5 USD per million input tokens, but 25 USD per million output tokens. Meaning: a chatty answer costs you disproportionately.

Lever 1: Prompt caching

The most effective lever for recurring context. If you send the same large block across many requests (a long system prompt, a fixed reference document, few-shot examples), you can cache that part. The provider stores the processed prefix; follow-up requests read it from cache instead of reprocessing it.

The saving is substantial: a cache read typically costs only a fraction of the normal input price (about one tenth at Anthropic, as of 2026). The first cache write is marginally more expensive (~1.25x at short lifetime) but pays off from the second request onward.

The condition: the cached part must be byte-identical and at the start of the prompt. A timestamp or a changing ID in the system prompt destroys the cache. Stable content up front, volatile content at the back.

Lever 2: Batch processing

If your tasks don’t need real-time answers — nightly classification of thousands of records, bulk summaries, offline analyses — use the Batch API. You send all requests at once and the result comes back asynchronously (usually within an hour, guaranteed within 24).

The payoff: a substantial discount, around 50 percent off all tokens at the common providers (as of 2026). For anything that tolerates latency, that’s the easiest cost halving there is.

Lever 3: Keep the context lean

The most common mistake: dumping too much into every prompt out of convenience. Half the wiki, the entire conversation history, five documents of which only one is relevant. Each of those tokens costs — on every single request.

The discipline: only include what this task needs. What’s too large or too rarely needed you fetch on demand via RAG (Retrieval-Augmented Generation) instead of carrying it permanently. Rather than forcing 50 pages into the prompt, you retrieve the two relevant paragraphs. This not only cuts cost but also improves answer quality (see lost in the middle in the context-window article).

Lever 4: The right model for the task

Not every task needs the most expensive model. A simple classification, a format conversion, a sentiment detection are handled just as well by a small, fast model — at a fraction of the cost.

As a dated snapshot (Anthropic, as of 2026), a small model sits at around 1 USD input / 5 USD output per million tokens, a top model at five times that. The rule of thumb: top model only where real reasoning depth matters. Routine tasks on the small model, demanding ones on the large. This split often saves more than all other levers combined.

Lever 5: Limit output length

Because output is pricier than input, answer length is a direct cost lever. Two measures:

  • Set max_tokens. A hard ceiling on the answer prevents runaway output. Caution: too low cuts off mid-response — set it realistically.
  • Ask for brevity in the prompt. “Answer in at most three sentences” or “just the table, no explanation” saves real output tokens.

A worked example

Suppose you process 10,000 support queries. Each has a 2,000-token system prompt (instructions, examples), 500 tokens of query, and produces a 300-token answer. With a model at 3 USD input / 15 USD output per million tokens:

  • Naive (no caching, synchronous): input 10,000 × 2,500 = 25M tokens → 75 USD. Output 10,000 × 300 = 3M → 45 USD. Total ~120 USD.
  • With system-prompt caching (read ~0.1x): the 2,000 stable tokens cost only one tenth. Input drops to roughly 10,000 × (200 + 500) = 7M effective → ~21 USD. Output unchanged at 45 USD. Total ~66 USD.
  • Plus batch (−50%): ~33 USD.

120 becomes 33 USD — with no loss of quality, purely through caching and batch. The numbers are illustrative and vary by provider and price snapshot; the ratio holds.

FAQ

Why is output more expensive than input? Generating text is more compute-intensive than reading it: each output token is produced one at a time, sequentially, while input can be processed in parallel. On current models the output price is often four to five times the input price — so short answers save disproportionately.

Is prompt caching always worth it? No. It’s worth it when the same large, stable context recurs across multiple requests. For one-off prompts with constantly changing content it does nothing — you’d only pay the marginal cache-write premium without ever collecting a read.

When should I use the Batch API? Whenever the task doesn’t need a real-time answer: nightly analyses, mass classification, offline summaries. The discount is around 50 percent. For interactive applications (chat) batch is unsuitable, because the answer comes back asynchronously.

Does a smaller model really save that much? For simple, well-defined tasks, yes. Classification, format conversion, or sentiment analysis are handled just as well by a small model at a fraction of the cost. You pay the premium for the top model only where real reasoning depth is needed.

How do I estimate my budget in advance? Multiply expected requests × (average input tokens × input price + average output tokens × output price). Many providers offer a token-counting endpoint that measures a prompt’s token count exactly, instead of guessing it.

Themenuebersicht

Arbeiten mit KIToken-Budget