Continuous batching: how one GPU serves a crowd

Decode is bandwidth-bound: generating one token streams the entire model’s weights through the GPU to produce a single word, leaving the maths units mostly idle. Speculative decoding spends that idle compute on one stream. Batching spends it differently and more fundamentally: serve many users at once, and let them all share the same trip over the weights.

It’s the single biggest lever for throughput when you’re serving more than one person, and the naive version leaves most of the gain on the table. The fix, continuous batching, is one of those ideas that’s obvious in hindsight and was genuinely missing from early serving stacks.

Why batching is nearly free

The roofline argument again. Loading the weights to decode one token costs a fixed amount of memory bandwidth. The crucial fact: loading them to decode eight tokens, one for each of eight different users’ sequences, costs almost the same bandwidth, because it’s the same weights loaded once and reused eight times. The maths units were idle anyway; now they’re doing eight tokens’ worth of work per load instead of one.

Throughput climbs with batch size (decode, one card)

Each extra sequence in the batch rides the same weight-load, so total throughput rises steeply at first. Eventually you exhaust the maths units or the memory for the KV cache and it flattens. Per-user speed dips a little; total served climbs a lot. Shape illustrative.

Note what batching does and doesn’t do: it raises total throughput dramatically, it does not make any single user’s reply faster (slightly slower, if anything). It’s a throughput trick, not a latency one. For serving a crowd, throughput is the bill.

The catch: requests don’t line up politely

So just gather requests into batches and go? That’s static batching, and it has an ugly failure mode the moment you serve real traffic: requests arrive at different times and finish at wildly different lengths. One user wants a three-word answer, another wants three thousand. Lock them into a fixed batch and the whole batch runs until the longest member is done.

Static batching: short replies sit idle waiting for the long one

A and B finished long ago but their slots stay locked, doing nothing, until C, the longest sequence, completes. New requests wait outside for the whole batch to clear. The GPU you paid for sits half-empty.

So short requests pay for long ones, finished slots sit idle, and fresh requests queue behind a batch that won’t clear. On bursty, mixed-length traffic, which is all real traffic, static batching wastes a lot of the card.

The fix: batch at the token, not the request

Continuous batching (also called in-flight batching) drops the idea of a fixed batch entirely. The insight: decode already runs one token-step at a time for every sequence, so treat each step as the unit, and let the batch’s membership change every step.

When a sequence emits its end-of-text token, it leaves the batch immediately, on that step, freeing its slot. No waiting for neighbours.
A waiting request joins the batch as soon as a slot opens (after a quick prefill of its prompt), mid-flight, without draining anything.

The batch becomes a living set: sequences flow in and out continuously, and the GPU stays as full as there’s work to fill it, step after step. A three-word answer is gone in three steps and its slot is instantly reused; the three-thousand-token answer never blocks anyone. That’s the whole idea, and it routinely multiplies real-world throughput several times over versus static batching on the same hardware, purely by not wasting slots.

Static batching

Dead simple to implement
Batch runs until the longest member finishes
Finished sequences hold idle slots
New requests wait for the batch to clear

Continuous batching

Slots freed and refilled every token-step
GPU stays full on bursty, mixed-length traffic
New requests join almost immediately
More bookkeeping, and KV memory must be managed

What it costs, and where it meets its limit

Continuous batching isn’t free engineering. Sequences entering and leaving every step means the scheduler is juggling the KV cache for a constantly-changing set of sequences, each of different length. That’s precisely why modern servers pair it with paged KV management (storing each sequence’s cache in fixed-size blocks, like virtual memory, so slots can be allocated and freed without fragmenting the card). The two ideas go together: continuous batching keeps the compute full, paged KV keeps the memory from becoming the thing that stops you.

And the ceiling is, as ever, the KV cache, not the maths. Every sequence in the batch needs its own cache, and on a single consumer card with limited memory left after the weights, that cache is what caps how many users you can hold at once. Throughput wants a bigger batch; memory says no. Tuning that trade, batch size against context length against free VRAM, is most of the art of serving an LLM on one card, and it’s where the 300 tok/s number is really won or lost the moment more than one person is asking.