Which Model Should You Use?

Match the model tier to the task rather than defaulting to the largest. A framework that survives the next model release.

Specific model recommendations expire within months. The decision framework does not, so this page covers the framework.

Tiers, not names

Every provider offers roughly three tiers, and the gap between them is large in both capability and price.

Small / fast. Cheapest, lowest latency. Handles classification, extraction, routing, simple formatting, and short summaries competently.

Mid. The general-purpose default. Most work belongs here.

Large / frontier. Best reasoning, highest cost, slowest. Justified for genuinely hard tasks.

The price spread across tiers is typically an order of magnitude or more. At any real volume, tier choice dominates every other cost decision you make.

Start small and move up

The instinct is to reach for the strongest model and stay there. That is expensive and often unnecessary.

Better approach: start with the small tier and escalate only on measured failure.

1. Try the small model on real examples from your actual task.
2. Measure whether it succeeds — with an eval, not an impression.
3. If it fails, move up a tier and measure again.

A surprising share of production work runs fine on small models. Classification, extraction, routing, and tagging rarely need frontier reasoning. Paying frontier prices for extraction is the most common avoidable AI cost.

What actually needs a big model

Multi-step reasoning where errors compound — agent loops especially, since reliability compounds downward and a weaker model’s per-step error rate destroys long runs.

Non-trivial code generation, particularly across multiple files or unfamiliar APIs.

Subtle judgment — nuanced tone, ambiguous edge cases, anything where “close enough” is not.

Long-context synthesis requiring genuine use of material spread across a large input.

Instruction adherence under complexity. Small models degrade first when many constraints interact.

Routing

You do not have to pick one model for everything.

Tier by task. Extraction on the small model, final response generation on the mid tier. This is the highest-value structural optimization available and it is underused.

Escalate on uncertainty. Run the small model, and if confidence is low or output fails validation, retry on a larger one. You pay for the big model only on the fraction of requests that need it.

Classify then route. A cheap first call decides difficulty, then dispatches accordingly. Worth it when your traffic is genuinely mixed.

The other axes

Capability is not the only dimension.

Latency. Interactive interfaces feel bad above a couple of seconds. A small model that answers immediately often beats a large one that answers better — streaming also changes perceived latency substantially.

Context window. If you send large documents, capacity is a hard filter. Note that advertised length does not guarantee good use of the middle.

Open weights versus hosted. Self-hosting means data never leaves your infrastructure and cost becomes fixed rather than per-token. It also means you operate GPUs. Worth it at high steady volume or under hard data-residency requirements; rarely worth it otherwise.

Data handling. Whether inputs may be retained or used for training is a contractual question, and it is often the deciding factor for regulated work.

Structured output support. If you need guaranteed schemas, constrained-generation support matters more than a few points of benchmark performance.

Benchmarks, cautiously

Public benchmarks are weak evidence for your specific task. They are aggregate, sometimes contaminated by training data, and never measure your prompt on your inputs.

Use them to narrow candidates, then test on your own examples. Twenty real cases from your actual workload tell you more than any leaderboard.

Estimating cost

Per request: input tokens plus output tokens, priced separately, output more expensive.

Two things dominate real bills:

Conversation history. Resent every turn, so cost grows with the square of length. Capping history is usually the single biggest lever.

Retrieved context. RAG chunks are input tokens on every call. Retrieving 20 chunks when 5 suffice costs four times as much, forever.

Measure on real traffic before optimizing. Estimates are consistently wrong, usually because of history growth.

What to remember

• Think in tiers, not model names — the framework outlives the releases.
• Start small, escalate on measured failure. Paying frontier prices for extraction is the common avoidable waste.
• Big models earn their cost on multi-step reasoning, real code generation, subtle judgment, and long-context synthesis.
• Route by task — cheap models for mechanical steps, expensive ones only where needed.
• Benchmarks narrow candidates; your own examples decide.

Next: Why Responses Stream