How Inference Serving Works
Batching, scheduling, and memory management. Why throughput and latency pull against each other, and what a serving system is actually doing.
On this page
Running a model for one request is straightforward. Serving thousands concurrently is an engineering problem with its own vocabulary, and understanding it explains most of what you observe from the outside — why latency varies, why long prompts are slow, why providers cap context.
Batching is the fundamental lever
Generating a token requires reading every active parameter from memory. That read dominates the cost, and arithmetic is comparatively cheap.
So processing one request wastes most of the hardware’s compute capacity. Process 32 requests together, read the parameters once, and serve all 32 for barely more than the cost of one.
Batching is therefore not an optimization but the core economics of serving. It is also the reason speculative decoding helps single users and not busy servers — it exploits idle compute that batching has already claimed.
Continuous batching
Naive batching waits for a full batch, runs all requests to completion, then starts the next. The problem is that requests finish at different times, so slots sit idle while the longest request finishes.
Continuous batching — sometimes called iteration-level scheduling — instead schedules per token step. When a request completes, a waiting request takes its slot immediately. The batch composition changes constantly.
This substantially raises utilization, and it is standard in modern serving systems. It also explains why your latency depends on other people’s traffic: your request shares each forward pass with whatever else is resident.
Prefill and decode compete
The two phases have opposite characteristics.
Prefill processes an entire prompt at once — highly parallel, compute-heavy, and quadratic in prompt length.
Decode generates one token per request per step — memory-bandwidth-bound and cheap per request.
Mixing them in one batch means a long prefill blocks decoding for every other request in that batch, producing visible stalls for users mid-response. Serving systems handle this by chunking prefill into pieces, or by running prefill and decode on separate hardware pools.
This is the mechanism behind an observation you may have made: someone else submitting a very long prompt can slow down your streaming response.
Memory is the binding constraint
Two things occupy memory: model weights, which are fixed, and KV cache, which grows with every token of every active request.
Cache usually dominates at scale, and it sets the concurrency limit. More concurrent requests need more cache; longer contexts need more per request.
Paged attention is the key technique. Rather than reserving contiguous memory for each request’s maximum possible length, allocate cache in fixed blocks on demand — the same idea as virtual memory. Requests that never reach their maximum do not waste the reservation, which allows far more concurrency on identical hardware.
Prefix sharing composes with it: requests sharing a leading prefix — the same system prompt, the same retrieved documents — can share those cache blocks. This is the mechanism behind prompt caching discounts, and the reason stable content belongs first in your prompt.
Throughput against latency
The central tension, and the reason no configuration is simply “best.”
Larger batches raise total tokens per second and increase per-request latency, since each request waits its turn in a bigger group. Smaller batches do the reverse.
Providers tune for a target — interactive endpoints favour latency, batch endpoints favour throughput. This is why batch APIs cost substantially less: they are permitted to fill large batches and schedule for efficiency rather than responsiveness.
What you observe from outside
Variable latency. Your request shares batches with other traffic.
Slow time-to-first-token on long prompts. Prefill is quadratic and must complete before generation starts.
Context limits below architectural capability. Concurrency times cache size hits memory before the model hits its length limit.
Rate limits in tokens, not just requests. Tokens, not requests, consume the constrained resource.
Cheaper batch tiers. Relaxed latency permits efficient scheduling.
Practical implications
Put stable content first so prefix caching can apply. Free, and the single easiest win.
Shorter prompts help more than you would expect, because prefill is quadratic.
Use batch endpoints when nothing is waiting on the result.
Expect latency variance and design for it — timeouts, retries, streaming so the user sees progress.
What to remember
- Batching is the economics of serving: parameters are read once for the whole batch.
- Continuous batching schedules per token step, so your latency depends on concurrent traffic.
- Prefill and decode compete; a long prompt from another user can stall your stream.
- KV cache is the binding memory constraint; paged attention and prefix sharing relieve it.
- Throughput and latency trade against each other, which is why batch tiers are cheaper.