Latency Optimization

Perceived speed is dominated by time to first token, and most of that is prefill. Where the milliseconds actually go.

Total response time is the wrong target. Users judge time to first token, because reading is slower than generation — once text starts flowing, the rest happens behind their reading.

So the work splits into two independent problems, and they have different fixes.

Where the time goes

Time to first token is dominated by prefill: processing your entire prompt before generation begins. It scales with prompt length, and steeply, since attention cost is quadratic.

Tokens per second is the decode loop — roughly constant per token, set by model size and server load.

Total time is the first plus output length divided by the second. Which one to attack depends on your shape: a long prompt with a short answer is prefill-bound, and shortening the prompt is the fix. A short prompt with a long answer is decode-bound, and only a smaller model or less output helps.

You cannot tell which you have without streaming and measuring both separately. This is the first thing to instrument.

Cutting time to first token

Shorten the prompt. The highest-leverage change, and better than linearly effective because prefill is quadratic. Retrieve five chunks instead of twenty; cap conversation history.

Enable prompt caching. Put stable content first so a cached prefix skips prefill entirely. Nearly free, and it is purely an ordering decision — see Cutting Your API Bill.

Start streaming immediately. Does not reduce actual latency and substantially reduces perceived latency, which is what you are optimizing.

Show progress before the model responds. For RAG, retrieval takes real time — display that step rather than a blank screen.

Cutting generation time

Smaller model. The largest single lever. Most tasks do not need the top tier, and routing by task is how you get speed without sacrificing the cases that need capability.

Ask for less output. Output tokens are generated sequentially, so a shorter answer is directly faster. “Answer in two sentences” plus a low max_tokens is a real latency change.

Skip reasoning where it does not help. Chain-of-thought and reasoning models trade latency for accuracy. Worth it on hard problems, pure cost on lookups.

Speculative decoding, if you control serving. Substantial gains for single requests, little under heavy batching.

Structural wins

Parallelize independent calls. Three retrievals or three classifications should run concurrently, not in sequence. Frequently the biggest available improvement in a multi-step pipeline, and frequently missed because the code reads naturally as a sequence.

Cache responses. A repeated question answered from your own cache has near-zero latency. Free where it applies.

Return early where possible. Show retrieved sources while generation runs; render partial structured output if the schema allows it.

Move work off the request path. Anything that does not need to happen before the response — logging, secondary classification, memory writes — belongs in a background task.

Multi-step and agent flows

For agent loops, each iteration is a full round trip, so latency is roughly steps times per-step time. Reducing step count matters more than making steps faster.

Two practical measures: run independent tool calls in parallel where the model requests several at once, and stream progress at the loop level — showing which tool is running — rather than leaving a twelve-step run silent.

Expect variance

Hosted APIs share hardware. Your latency depends on concurrent traffic, and another user’s long prompt can stall your stream mid-response.

Design for it: generous timeouts, retries with backoff, streaming so partial output is visible, and track percentiles rather than averages. The tail is what users experience and complain about; an average hides it completely.

What to remember

• Optimize time to first token — reading is slower than generation, so perceived speed is dominated by the start.
• Prefill (quadratic in prompt length) drives TTFT; decode drives the rest. Measure both separately before optimizing.
• Shorten prompts, enable prefix caching, and stream immediately.
• For generation: smaller model, less output, skip unnecessary reasoning.
• Parallelize independent calls — often the largest win in a pipeline — and move non-essential work off the request path.
• Track percentiles, not averages; hosted latency varies with other people’s traffic.

Next: Reliability Patterns