Cost Monitoring and Attribution

Token spend is easy to measure and easy to ignore until the bill arrives. What to track, and which dimension actually explains your costs.

Every response includes token counts. Logging them costs nothing. Most teams do not, and discover their cost structure from a monthly invoice that attributes nothing.

Attribute by feature, not by total

A total spend figure tells you there is a problem, not where. The useful unit is cost per feature per request.

Tag every model call with the feature that made it — summarization, classification, chat, query rewriting — then aggregate. The distribution is consistently surprising: a feature nobody thought about frequently dominates, usually because it runs on every page load or resends history.

Add two more dimensions once feature attribution exists:

Per user or tenant. Reveals whether spend is broad or concentrated in a handful of heavy users, which determines whether the fix is optimization or rate limiting.

Per model. Confirms whether traffic is landing on the tier you intended. Calls silently defaulting to an expensive model is a common and invisible waste.

The three things that dominate bills

Conversation history. Because the model has no memory between calls, each turn resends the full transcript, so total tokens grow with the square of conversation length. This is the most common surprise, and it is invisible per request — each individual call looks reasonable.

Track average conversation length and tokens per conversation, not just tokens per request.

Retrieved context. RAG chunks are input tokens on every call. Retrieving twenty where five suffice costs four times as much, permanently. Track chunks retrieved per query as a cost metric.

Model tier. Tier prices differ by an order of magnitude, so routing decisions swamp every other optimization.

Metrics worth a dashboard

Cost per hour, with a threshold alert. The classic unmonitored failure is a loop or a runaway prompt found on the monthly bill; hourly tracking catches it in an hour.

Cost per successful outcome, not per request. A cheap call that fails and retries three times is not cheap. This is the number that actually matters and the one least often computed.

Input-to-output ratio. A rising ratio means prompts are growing — usually history or retrieved context creeping up.

Cache hit rate, where prompt caching applies. A drop means something destabilized the prefix, often a timestamp or a per-request value that migrated to the top of a prompt.

Tokens per feature over time. Prompt changes shift cost, and nobody notices without a trend.

Set limits in code

Monitoring tells you what happened. Limits prevent it.

Cap max_tokens on every call. Not a target, a ceiling against runaway generation.

Bound input length before spending a call on it.

Rate limit per user, not only globally.

Cap agent iterations and total tokens per agent run — cost grows quadratically with steps, so an uncapped loop is an uncapped bill.

Use provider budget alerts where available, as a backstop rather than a primary control.

Before optimizing

Measure first. Two weeks of attributed data changes what you would have optimized.

Then take the levers in order of size: model tier, then conversation history, then prompt caching via stable-content-first ordering, then retrieved chunk count, then output length.

And evaluate quality alongside cost. A change that halves spend and breaks a feature is not a win, which is why evals belong in the same loop as cost work.

What to remember

• Attribute by feature, then by user and model — totals identify a problem, attribution locates it.
• The three dominant costs: quadratic conversation history, retrieved context volume, and model tier.
• Track cost per successful outcome, hourly spend with alerts, input-to-output ratio, and cache hit rate.
• Enforce caps in code: max tokens, input bounds, per-user rate limits, agent iteration and token budgets.
• Measure for two weeks before optimizing, and check quality alongside cost.

Next: Latency Optimization