Cutting Your API Bill

Four places cost hides, ordered by how much they usually save. Most bills are dominated by one fixable thing.

Measure before optimizing. Log input and output tokens per request, grouped by feature, and the answer is usually obvious and usually not what you guessed.

The four levers below are ordered by how much they typically save.

1 · Model tier

Almost always the largest factor. Tier prices differ by an order of magnitude or more, so moving a high-volume task down a tier beats every other optimization combined.

The pattern that works: start with a small model and escalate only on measured failure. Extraction, classification, and routing rarely need a frontier model, and paying frontier prices for them is the most common avoidable cost in production.

Better still, route by task — cheap models for mechanical steps, expensive ones only where reasoning genuinely matters.

2 · Conversation history

The quiet budget killer. Because the model has no memory between calls, every turn resends the entire transcript. Total tokens across an n-turn conversation grow with n², not n.

A twenty-turn conversation is not twice a ten-turn one. It is roughly four times.

Fixes:

Cap history deliberately. Keep the system prompt plus the last several turns. Decide this yourself rather than letting a framework silently truncate.

Summarize older turns. Replace early history with a compact summary. Loses detail, extends the usable conversation.

Start fresh when the topic changes. Users rarely need turn 3 when discussing something unrelated at turn 30.

3 · Prompt caching

Providers can cache the processed form of a prompt prefix. Repeated prefixes then skip prefill and are billed at a substantial discount.

The mechanism is KV cache reuse: identical leading tokens produce identical keys and values, so they can be computed once and reused.

This gives one concrete structural rule: stable content first, variable content last.

[system prompt]        ← fixed, cached
[few-shot examples]    ← fixed, cached
[retrieved documents]  ← changes per query
[user question]        ← changes per query

Reverse that order and you cache nothing. It is the cheapest optimization available, and it is purely about ordering.

Two caveats: the cache is exact-prefix, so a single changed character early invalidates everything after it — never put a timestamp at the top of a system prompt. And caches expire quickly, so benefit depends on request frequency.

4 · Token volume

Smaller inputs and outputs cost less, and output tokens cost more than input.

Cap max_tokens at what you need. “Answer in two sentences” plus a low cap is real money at volume.

Retrieve fewer chunks. RAG context is input tokens on every call. Twenty chunks where five suffice costs four times as much forever — and reranking usually makes five better than twenty anyway. Cheaper and more accurate at once.

Trim few-shot examples once you know which ones carry weight. Every example is tokens on every request.

Skip reasoning where it does not help. Chain-of-thought multiplies output tokens, which are the expensive kind. Worth it for hard reasoning, wasteful for lookups and formatting.

Two structural options

Response caching. If the same question recurs, cache the answer in your own layer and skip the model entirely. Free where it applies. Exact-match caching is trivial; semantic caching — matching near-duplicate questions by embedding similarity — catches more and risks serving an answer to a subtly different question.

Batch APIs. Several providers offer substantial discounts for asynchronous processing with a delayed completion window. If nothing is waiting on the result — nightly classification, bulk enrichment — this is a large saving for no quality cost.

What not to do

Do not compress prompts into unreadable shorthand. Savings are small and quality loss is real.

Do not drop structured output constraints to save tokens. Parse failures and retries cost more than the schema did.

Do not over-summarize history in ways that break the conversation. Users repeating themselves generates more requests than you saved.

Do not optimize before measuring. The distribution of cost across features is consistently surprising.

What to remember

• Measure per-feature token counts first; the biggest cost is rarely where you expect.
• Model tier is the largest lever, then conversation history — which grows quadratically and is usually the hidden problem.
• Prompt caching is nearly free: stable content first, variable content last, and never a timestamp at the top.
• Fewer retrieved chunks is both cheaper and usually more accurate.
• Response caching and batch APIs eliminate cost entirely where they apply.

Next: How Do You Know It Works?