Main area
Agency
ki-orchestrierung

LLM Routing and the LLM Router

Redaktion ·

LLM routing just means this: not every request goes to the same model. A small gatekeeper — the router — looks at what comes in and decides which model should handle it. Easy question? Off to a cheap, fast model. Tricky one? Off to the big, expensive one. And that’s exactly where it clicks: you stop paying the premium price for every trivial thing.

Why it exists at all

Models within one family are often far apart on price. A small model costs only a fraction per million tokens of what the top model charges — the gap can easily be 15x to 60x. So if you stubbornly send every single request to the most expensive model, you’re burning money on tasks a smaller model would have handled just as well.

The thing is, most requests in a real product aren’t hard at all. A quick rephrase, a simple classification, a standard reply — none of that needs a heavyweight. Only a slice of requests is genuinely demanding. The router separates those two groups before a single cent flows to the big model.

How a router decides

At its core, a router is a classifier. It takes the request and estimates how hard it is — or more precisely, whether the strong model would be noticeably better here than the weak one. If that probability sits above a threshold you set, the request goes to the strong model. Below it, the cheap one is enough.

In practice there are three common ways to do this:

  • Rule and keyword routing. Fixed rules decide based on keywords or patterns. If “legal question” or “code review” shows up, it goes to the strong model. It sounds crude, but it’s transparent and fast.
  • Embedding-based (semantic) routing. The request is turned into an embedding — meaning a numeric representation of its meaning — and compared against stored example paths. The closest match wins. This handles requests that don’t match your rules word for word.
  • LLM-based routing. A small model reads the request and decides where it goes itself. Flexible, but slower and pricier than the other two — so it’s often kept just as a fallback.

A good semantic router makes its call in under half a second. That matters, because every millisecond the router spends gets added on top before the actual model even starts working.

What you save

The real-world numbers are solid. A trained router from RouteLLM held about 95 percent of the strong model’s quality while passing only roughly 14 to 26 percent of requests through to the expensive model. The bottom line was 75 to 85 percent lower cost compared to the blunt “always use the top model” approach.

Where it gets tricky

A router is only as good as its judgment. Classify a hard request as easy, and it lands on the weak model — and the user gets a thin, maybe wrong answer. Classify an easy request as hard, and you pay the premium for nothing. Both happen, and both have consequences.

Semantic routing in particular can miss the mark when requests sound alike but actually want something quite different. That’s why two things belong in any serious routing setup: a fallback path (when the router gets out of its depth, the request goes to the strong model to be safe) and ongoing measurement of the hit rate. Routing isn’t something you set once and forget — you’ll want to retune the threshold against your real traffic.

If you want to go deeper on picking models, see the entry on comparing AI models, and for the cost side AI pricing explained.

FAQ

What’s the difference between an LLM router and multi-agent orchestration?
A router picks which single model handles a request — one decision, one model. Multi-agent orchestration lets several agents work on a task together and splits it up. So routing is a selection question, orchestration is a division question. They don’t rule each other out: often a router sits in front of the individual agents.
Do I need an extra model to do my own routing?
Not necessarily. Rule and keyword routing need no model at all. Semantic routing needs only a cheap embedding model, not a second language model. Only LLM-based routing brings in an additional small model as the decider.
How much can I realistically save?
That depends heavily on how many of your requests are genuinely easy. In well-documented cases, savings landed between 30 and 85 percent versus “always the strongest model.” The bigger the price gap between your models and the higher the share of easy requests, the more you get out of it.
What happens when the router decides wrong?
In the bad case, the user gets a weaker answer because a hard request was wrongly sent to the small model. That’s why you build in a fallback path: on uncertainty or a poor answer, it automatically escalates to the strong model. Alongside that, you measure the hit rate continuously and retune the threshold.
Is routing even worth it for small projects?
With low traffic and consistently demanding requests, the effort often isn’t worth it — you just pick the right model directly. But as soon as you have lots of requests with mixed difficulty, routing starts to pay off noticeably.

Themenuebersicht

KI-OrchestrierungKeyword-Routing