Continuous Batching: The Trick Behind High-Throughput LLM Serving
Continuous batching admits and retires requests within an in-flight batch instead of waiting for fixed batches to complete, raising GPU utilization and throughput for LLM serving.
If you have wondered how modern inference servers squeeze so many concurrent users out of a single GPU, continuous batching is a large part of the answer. It is a scheduling technique that keeps the GPU saturated by adding new requests to a running batch and retiring finished ones, all without pausing to assemble a fresh batch from scratch. Compared with the older static approach, it dramatically raises utilization and throughput, which is why nearly every high-performance serving stack uses it.
The problem with static batching
In static batching, the server collects a fixed group of requests, runs them all together until the slowest one finishes, then starts the next group. The trouble is that requests in a batch produce different output lengths. A request that needs five tokens finishes long before one that needs five hundred, but in a static batch it must wait for the longest one before its slot is freed. The GPU spends much of its time processing padding and idle slots, and new requests queue up behind a batch that is mostly done.
| Aspect | Static batching | Continuous batching |
|---|---|---|
| Batch composition | Fixed until all finish | Changes every step |
| GPU utilization | Drops as requests finish | Stays high |
| New request admission | Waits for next batch | Joins almost immediately |
| Throughput | Lower | Higher |
How continuous batching works
Continuous batching, sometimes called in-flight batching, operates at the granularity of a single generation step rather than a whole request. Because language models generate one token per step across the whole batch, the scheduler gets a decision point on every step.
- On each step, the server produces the next token for every active request in the batch.
- Any request that just produced its final token is removed and its result returned immediately.
- Waiting requests are admitted into the freed slots, often starting their prompt processing right away.
- The batch composition therefore changes continuously, always staying as full as possible.
The effect is that the GPU rarely processes empty or finished slots. A short request leaves the moment it is done, and a queued request takes its place at once, so the hardware stays busy with useful work.
Why throughput rises so much
Two wins stack up. First, finished requests no longer hold their slots hostage waiting for the longest request, eliminating a major source of wasted computation. Second, queued requests start almost immediately instead of waiting for the next batch boundary, which also lowers their time to first token. The result is both higher throughput and, often, better tail latency at the same time, which is unusual since those goals normally conflict.
Interaction with the KV cache
Continuous batching shines when paired with memory-efficient cache management such as paged attention. Because requests join and leave constantly, the server needs to allocate and free KV cache memory dynamically and without heavy fragmentation. Paged allocation makes that practical, which is why the two techniques usually appear together in modern serving stacks.
What it does and does not solve
Continuous batching is mainly a throughput and utilization win. It is not a magic latency reducer for a single isolated request, since one request alone cannot benefit from better batch packing. Its value appears under concurrency, where many requests of varying lengths arrive together. A few points to keep in mind.
- It raises throughput most when output lengths vary widely across requests.
- It improves first-token latency for queued requests by admitting them sooner.
- It depends on dynamic cache memory management to avoid fragmentation.
- It does not remove the underlying throughput versus latency tradeoff, but it shifts the curve favorably.
What to look for in a serving stack
When evaluating an inference server or a managed endpoint, continuous batching should be standard. Confirm that it is enabled, that it pairs with paged or otherwise efficient cache handling, and that you can set sensible limits on maximum concurrency and context length so the scheduler does not oversubscribe memory. Then load test with a realistic mix of short and long requests, because that mix is exactly where continuous batching earns its keep and where a static-batching system would fall behind.
Tuning the scheduler
Continuous batching is not entirely hands-off. A few knobs shape how it behaves under your specific traffic, and tuning them keeps the GPU efficient without breaking latency targets.
- Maximum batch size or token budget: a cap on how many requests or tokens the scheduler packs at once. Higher caps raise throughput but can push tail latency up, so set it against your latency target.
- Maximum context length: bounding the longest allowed request protects the cache from a single huge prompt that would starve everyone else.
- Admission and queueing policy: deciding how aggressively to admit waiting requests influences first-token latency for newcomers versus steady throughput for active ones.
- Preemption behavior: some stacks can pause and resume long requests to let shorter ones through, trading a little overhead for fairer latency.
Why it pairs with autoscaling
Because continuous batching raises how much work a single replica can absorb, it changes your scaling math for the better. Each replica handles more concurrent users before it saturates, so you reach for additional GPUs later and run fewer, busier replicas overall. That higher per-replica capacity translates directly into lower cost per token, since the fixed cost of each GPU is spread across more useful work. When you combine continuous batching with autoscaling, you get a system that both packs each replica tightly and adds replicas only when genuinely needed, which is close to the efficient frontier for serving variable traffic.
Conclusion
Continuous batching is the scheduling trick that lets a single GPU serve far more concurrent users than static batching ever could. By making admission and retirement decisions on every generation step, it keeps the batch full, frees finished requests instantly, and admits waiting ones immediately, raising both throughput and first-token latency at once. It works best under varied, concurrent load and depends on efficient KV cache management to avoid fragmentation. For anyone serving language models at scale, it is not an optional optimization but the baseline expectation, and any serving stack worth using should have it on by default.