Smart Model Routing: Right Task, Right Model, Right Cost

The default enterprise AI architecture is one chatbot pointed at everything, and it fails in both directions at once. Routine work gets processed at frontier prices, and crucial decisions get whatever the general-purpose bot produces. The fix is the same one operations discovered a century ago: match the tool to the task.

Smart model routing treats models as a portfolio. Crucial tasks go to a top-tier model. High-volume work goes to efficient open weights you host yourself. The bulk of small tasks goes to small local models that cost almost nothing. A router assigns every request, and confidence thresholds escalate anything the cheap tier should not decide alone.

The architecture

Tier 1 is frontier: Claude and GPT-class models reserved for architecture-level decisions, complex multi-step reasoning, disputes, and anything high stakes and customer-visible. This tier is deliberately expensive and deliberately rare, and it carries a human checkpoint.

Tier 2 is the workhorse: strong open-weight models, Kimi K2, GLM, Llama 70B class, self-hosted in your own cloud. They handle high-volume summarization, long-document processing, retrieval-augmented answers, and anything touching data that cannot leave. At sustained volume they run for cents on the frontier dollar, and inside a zero-egress enclave the sensitive-data question answers itself.

Tier 3 is small and local: task-tuned 8B-class models doing classification, extraction, templated drafting, and the routing itself, at near-zero unit cost and millisecond latency. In most estates this tier quietly absorbs the majority of requests.

Around all three sit purpose-built agents rather than one bot for everything: each with one job, scoped tools, guardrails, evaluation gates, and logging. An agent that only files claims cannot be talked into discussing pricing.

Why routing beats standardizing

Three reasons. Economics: token spend concentrates where capability actually matters, and the cheap tiers carry the volume. Quality: crucial tasks stop competing with bulk work for the same generic bot, and each agent is evaluated against its own job. Sovereignty: the model portfolio stays replaceable, so when a better or cheaper model ships, you swap a tier, not a platform. The selection math is the capability-per-dollar benchmarking we run in private LLM hosting engagements, on your tasks rather than public leaderboards.

An illustrative build: claims operations

A composite drawn from how we design these systems, with rounded numbers for clarity. A mid-market insurer's claims unit receives roughly a million documents a month. Tier 3 classifies every document and extracts standard fields, resolving about four in five automatically. Tier 2, an open-weight model in a zero-egress enclave, summarizes the remainder, drafts adjuster notes, and answers policy questions against internal documents, since claims files cannot leave the tenant. Tier 1 sees only the hard residue: disputed liability, coverage exceptions, and anything headed for litigation, a low single-digit share of volume, each with an adjuster checkpoint.

The economics invert the naive design: the overwhelming majority of tokens run on tiers that cost little, frontier spend concentrates on the small set of decisions worth it, and the audit trail shows which model touched what, under which permissions, for every claim.

What it takes to run

The router is policy plus a small model, not magic: criticality, sensitivity, volume, latency, and budget as explicit rules, confidence-based escalation, and evaluation on every tier so promotions are earned. Governance wraps the whole thing, mapped to ISO 42001 and the NIST AI RMF, which is the discipline our enterprise AI consulting practice builds in from the first sprint. The data side usually lands on Databricks or Palantir, governed either way.

One model for everything is a procurement decision. A routed portfolio is an architecture. Enterprises that make the switch stop debating which model is best and start asking the better question: best for which task, at what price, under whose controls.