LLM Model Routing: Cut AI Costs Without Losing Quality

AI on a Budget. This is part of our running series on cutting AI costs. See the full series →

Here is an uncomfortable truth about your AI bill: you are paying Rolls-Royce prices to answer questions a bicycle could handle. “Reformat this list.” “What’s the capital of Peru?” “Fix this typo.” You route every one of those to the same expensive frontier model you use for gnarly reasoning, and you pay accordingly. Model routing is the fix, and it is one of the best-kept value plays in our AI-on-a-budget series. Done right it cuts inference costs by 40 to 85% while keeping quality near enough identical to always using the top model.

Table of Contents

The core idea: cheap first, escalate only when needed

Routing means putting a traffic controller in front of your models. Each request gets sized up, then sent to the cheapest model that can actually handle it. Only the genuinely hard stuff gets escalated to the pricey frontier model. There are two flavours worth knowing:

Pre-generation routing: a lightweight classifier looks at the request before any model runs and picks the right one. Fast, no wasted calls.
Post-generation cascading: the cheap model answers first, then a confidence check decides whether the answer is good enough or needs bumping up to the big model. Slightly more compute, but it catches edge cases the classifier misses.

The research behind this is not hand-wavy. RouteLLM, the open-source framework from LMSYS, reported over 85% cost reduction on the MT-Bench test while still delivering around 95% of GPT-4’s quality, by sending the easy majority to a cheaper model. The older FrugalGPT paper pushed cascading to a headline 98% cost reduction on some workloads. Your mileage depends on how skewed your traffic is, and for most real apps it is very skewed towards easy.

The maths of why this works

Think about the price gap. A frontier model might cost $3 to $15 per million tokens. A budget-tier model like Gemini Flash-Lite sits around $0.25 input and $1.50 output per million, and strong open models on cheap hosts go lower still. If 80% of your queries can be served by a model that is 10 to 20 times cheaper, and only 20% need the expensive brain, your blended cost collapses even though your hardest questions still get top-tier answers. You are not downgrading quality, you are refusing to overpay for the trivial 80%.

How to set it up without a research team

Try RouteLLM as a drop-in. It acts as a replacement for the OpenAI client or runs as an OpenAI-compatible server. You set a strong model and a weak model with --strong-model and --weak-model flags, pick the pre-trained mf router, and it decides per query. Minimal code change.
Or roll a simple rule-based router. You do not need machine learning to start. Route by task type: short factual and formatting jobs to the cheap model, anything involving multi-step reasoning, code architecture or long context to the expensive one. Even a crude keyword-and-length heuristic captures most of the saving.
Use a gateway. Tools like LiteLLM sit in front of multiple providers and let you swap models behind one endpoint, which makes routing and fallbacks far less painful.
Measure the escalation rate. Watch what fraction of queries get bumped to the expensive model. If it is over 50%, your cheap model is too weak or your threshold too jumpy. Tune until most traffic settles on the cheap tier.

The trap to avoid

Do not route blind and hope. The one way routing bites you is silently sending a hard question to a model that fumbles it and returning a confident wrong answer. That is why cascading with a confidence check beats naive routing for anything high-stakes: the cheap model gets a go, but a weak or low-confidence answer triggers the escalation automatically. Set the threshold conservatively at first, watch your quality metrics, then loosen it as you gain trust. The goal is a bill that looks like you used the cheap model and output that looks like you used the expensive one.

Frontier models are extraordinary. They are also wasted on “make this bullet list into a sentence.” Put a router in front, and let the expensive brain do expensive work.

Did you know: the same escalate-only-when-needed logic runs your phone network. Calls are handled by the cheapest available path and only bumped to costlier routes when the cheap ones are congested, a technique telecoms engineers have called least-cost routing since the 1980s.

Sources:

The free stack. One email a week: the AI tools and moves that actually matter, hype filtered out. Subscribe free →