Main area
Agency
ki-konzepte

Working with the LLM API — Streaming, Caching, Rate Limits

Redaktion ·

Why this article — what pricing doesn’t tell you

If you’ve ever pushed an LLM API into production, you know the pattern. The prototype works in two hours with requests.post(...). Four weeks later the UX feels sluggish because every reply takes 8 seconds. Your bill is double what you planned because the 18,000-token system prompt gets paid for on every single call. And under load you start seeing 429s with nothing actually broken in the backend.

The pricing primer covers what an API call costs. This article covers how to use the API cleanly — the four mechanics that separate a throwaway script from a production integration: streaming, prompt caching, batch processing, and rate limits.

Streaming — why answer token by token

LLMs generate output one token at a time. Without streaming, the client waits until the model is done and then receives the entire text at once. For 600 output tokens on a Sonnet-class model, that’s 8–12 seconds of silence — and users in that window are wondering whether something is broken.

With streaming, the server sends each token (or small token group) the moment it’s generated. Perceived latency drops to the time-to-first-token — typically 200–800 ms — and human reading speed never catches up to generation speed anyway.

How it actually works

Both OpenAI and Anthropic stream over Server-Sent Events (SSE) — a simple HTTP standard where the response is a long sequence of data: {...}\n\n lines. You enable it via stream: true:

stream = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain RAG to me."}],
    stream=True,
)
for event in stream:
    if event.type == "content_block_delta":
        print(event.delta.text, end="", flush=True)

In the browser you either use the EventSource API (read-only, GET) or your own fetch reader (POST with a streaming body). Frameworks like the Vercel AI SDK or LangChain abstract the details.

When streaming doesn’t help

  • Structured output / JSON mode. You need the complete JSON before you can validate or process it. Streaming JSON parsers exist, but they’re nasty.
  • Backend pipelines without a UI. A classification pipeline writing to a database has no audience. Streaming just adds complexity.
  • Tool-use loops. If the model stops after 50 output tokens to call a tool, latency is already short — streaming buys little.

Rule of thumb: stream if a human is watching. Otherwise don’t.

Prompt caching — the biggest cost lever besides switching models

Every API call ships the same things: system prompt, tool definitions, maybe a 50k-token document as context. The model has to push those tokens through its internal layers every time, and you pay for them every time. Prompt caching stores those internal representations server-side and lets you reuse them as long as the prefix stays bit-identical.

Anthropic — explicit, with breakpoints

Anthropic makes you mark what to cache via cache_control breakpoints (max. 4 per request). Minimum size is model-dependent: 1,024 tokens for Sonnet 4.x, 4,096 tokens for Opus 4.7 and Haiku 4.5. Below that, caching silently doesn’t happen — so always check cache_creation_input_tokens and cache_read_input_tokens in the response.

Pricing structure (relative to base input):

  • Cache write (5 min TTL): 1.25×
  • Cache write (1 h TTL): 2.0×
  • Cache read: 0.1×

Concrete: a 30,000-token context queried 50 times an hour costs without cache 50 × $3 × 0.03 = $4.50. With 1-hour cache: one write at 2.0 × $3 × 0.03 = $0.18, plus 49 reads at 0.1 × $3 × 0.03 = $0.44 → $0.62. 86 % saved.

Breakpoint order is fixed: tools → system → messages. Any change higher up invalidates everything below. Lookback window: 20 blocks.

OpenAI — automatic, with conditions

OpenAI caches automatically, no markup required. Conditions: prompt ≥ 1,024 tokens, exact prefix match in 128-token increments. Cache hits give a 50 % discount on input price and reduce latency. There’s no 1-hour variant — the cache window is a few minutes.

Anti-patterns that kill your cache hit rate

  • Timestamp in the system prompt. “Current time: 2026-05-04T14:23:11Z” → hash changes every second, hit rate zero.
  • Randomly ordered tool definitions. Non-deterministic JSON serialization breaks the prefix match.
  • Variable user ID at the top. Personalization belongs at the end, not in the cache prefix.
  • Cache breakpoint after the variable part. The breakpoint must sit on the last unchanging block — not after the user question.

Batch API — 50 % off for asynchronous workloads

Both OpenAI and Anthropic offer a Batch API: you upload a JSONL file with hundreds or thousands of requests, the provider processes them asynchronously, and bills you 50 % less than the realtime API. SLA window: up to 24 hours, but in practice usually 1–6 hours.

What it’s built for

  • Embedding generation across an entire knowledge base
  • Classification or categorization of large corpora
  • Synthetic data generation for fine-tuning
  • Backfill analyses, nightly reports

The format is trivial — one line per request, each request has a custom_id that you find again in the result file:

{"custom_id": "doc-001", "method": "POST", "url": "/v1/messages", "body": {"model": "claude-haiku-4-5", "max_tokens": 200, "messages": [{"role": "user", "content": "Classify: ..."}]}}
{"custom_id": "doc-002", "method": "POST", "url": "/v1/messages", "body": {"model": "claude-haiku-4-5", "max_tokens": 200, "messages": [{"role": "user", "content": "Classify: ..."}]}}

What you get — and what you don’t

Batch comes with separate, higher rate limits (Anthropic Tier 1 allows up to 100,000 in-flight batch requests). Caching and batch discounts stack on OpenAI — cached tokens inside a batch run at just 25 % of list price.

What you don’t get: realtime. If a user is sitting in front of a UI waiting for an answer, batch is the wrong choice. There’s no streaming inside batch either — you get the full answer at once when the job completes.

Rate limits — TPM, RPM, and the 429 reflex

Providers limit you in several dimensions at once:

  • RPM — requests per minute (counts every call regardless of size)
  • ITPM — input tokens per minute
  • OTPM — output tokens per minute
  • on OpenAI sometimes folded into a combined TPM

Hit any of those and you get an HTTP 429 with a retry-after header (in seconds). Anthropic additionally returns anthropic-ratelimit-*-remaining headers on every successful response — so you can throttle before you hit the wall, not after.

How the limit works under the hood: token bucket

Anthropic states it explicitly, OpenAI implements something similar: a token-bucket algorithm. Picture a bucket with capacity N; every request takes a token out; the bucket refills continuously. Consequence: bursts are allowed as long as the average holds — but 60 RPM doesn’t mean “60 in the first second, then nothing,” it means roughly “1 per second on average.”

Tier structure (Anthropic example)

| Tier | Requirement | Sonnet 4.x ITPM | OTPM | RPM | |---|---|---|---|---| | 1 | $5 deposit | 30,000 | 8,000 | 50 | | 2 | $40 cumulative | 450,000 | 90,000 | 1,000 | | 3 | $200 cumulative | 800,000 | 160,000 | 2,000 | | 4 | $400 cumulative | 2,000,000 | 400,000 | 4,000 |

Tier upgrades are automatic once you cross the threshold — no application form. OpenAI has a similar five-step tier structure.

Cache reads usually don’t count against ITPM

A subtle but important point: on current Anthropic models, cache_read_input_tokens don’t count against your ITPM limit. With an 80 % cache hit rate on a 2M ITPM tier, you effectively process 10M input tokens per minute. Caching isn’t just a cost lever — it’s a throughput lever.

Backoff — exponential, with jitter

Reacting to a 429 with “try again immediately” only escalates the problem. Standard pattern:

  1. On the first 429, respect the retry-after value.
  2. If the header is missing, back off exponentially: 1s → 2s → 4s → 8s → 16s.
  3. Add jitter (random ±20 %) so a hundred parallel workers don’t reconverge in lockstep.
  4. Cap retries at 5–6, then fail cleanly and let the calling code handle it.

The OpenAI and Anthropic SDKs ship backoff out of the box — usually relying on that beats rolling your own.

Production stack: what actually plays together

A typical production setup combines all four mechanics:

  1. System prompt + tool definitions + long context in one block, with cache_control (1-h TTL for frequently repeating queries, 5-min otherwise).
  2. Streaming on for the UI so time-to-first-token stays under 1 s.
  3. Bulk jobs (nightly re-embedding, corpus classification) into the Batch API, not the sync API.
  4. Leave SDK backoff enabled, choose a conservative worker count, monitor anthropic-ratelimit-tokens-remaining and throttle actively below 10 %.

That typically removes 50–80 % of token cost, keeps the UX fluid, and limits 429s to a handful per day.

FAQ

Is caching worth it for short prompts?
No. Below the minimum size (1,024 / 4,096 tokens depending on model) caching just doesn't happen. For prompts barely over the threshold, break-even is at 3–5 reuses.
Can I stream and cache at the same time?
Yes. They're independent. Cache reads happen before the first output token, streaming after.
What about gpt-realtime / voice APIs?
Different world — bidirectional WebSocket streaming, separate pricing, separate limits. Not covered here.
How do I know my cache is working?
The response usage object exposes cache_read_input_tokens and cache_creation_input_tokens. If both are zero, the cache didn't hit — check your prefix.

Conclusion

Streaming, caching, batch, and rate limits aren’t advanced optimizations — they’re table stakes. Skip them and you end up with sluggish UX, an expensive bill, or an integration that buckles under load. Usually all three at once.

The order to tackle them is clear: streaming first (UX pain is visible immediately), caching second (biggest cost lever), batch third (once asynchronous workloads appear), rate limits fourth (once load arrives). If you’ve internalized the pricing model, these four mechanics are the next thing to nail down. Everything beyond that is detail.

Themenuebersicht

KI-KonzepteAPI-Nutzung