# Continuous batching: how one GPU serves a crowd > A single decode stream wastes most of the GPU. The fix is to serve many requests at once and let them share each pass over the weights, but only if you stop waiting for the whole batch to finish. Continuous batching, and why it's the throughput trick that matters. Source: https://rfriedmann.de/blog/continuous-batching/ Published: 2026-06-24 · Track: log · Level: Advanced [Decode is bandwidth-bound](/blog/prefill-vs-decode-roofline/): generating one token streams the entire model's weights through the GPU to produce a single word, leaving the maths units mostly idle. [Speculative decoding](/blog/speculative-decoding/) spends that idle compute on one stream. Batching spends it differently and more fundamentally: serve *many users at once*, and let them all share the same trip over the weights. It's the single biggest lever for throughput when you're serving more than one person, and the naive version leaves most of the gain on the table. The fix, continuous batching, is one of those ideas that's obvious in hindsight and was genuinely missing from early serving stacks. ## Why batching is nearly free The roofline argument again. Loading the weights to decode one token costs a fixed amount of memory bandwidth. The crucial fact: loading them to decode *eight* tokens, one for each of eight different users' sequences, costs almost the same bandwidth, because it's the same weights loaded once and reused eight times. The maths units were idle anyway; now they're doing eight tokens' worth of work per load instead of one.
Throughput climbs with batch size (decode, one card)
[diagram omitted — see the page for the chart]
Each extra sequence in the batch rides the same weight-load, so total throughput rises steeply at first. Eventually you exhaust the maths units or the memory for the KV cache and it flattens. Per-user speed dips a little; total served climbs a lot. Shape illustrative.
Note what batching does and doesn't do: it raises *total* throughput dramatically, it does **not** make any single user's reply faster (slightly slower, if anything). It's a throughput trick, not a latency one. For serving a crowd, throughput is the bill. ## The catch: requests don't line up politely So just gather requests into batches and go? That's **static batching**, and it has an ugly failure mode the moment you serve real traffic: requests arrive at different times and finish at wildly different lengths. One user wants a three-word answer, another wants three thousand. Lock them into a fixed batch and the whole batch runs until the *longest* member is done.
Static batching: short replies sit idle waiting for the long one
[diagram omitted — see the page for the chart]
A and B finished long ago but their slots stay locked, doing nothing, until C, the longest sequence, completes. New requests wait outside for the whole batch to clear. The GPU you paid for sits half-empty.
So short requests pay for long ones, finished slots sit idle, and fresh requests queue behind a batch that won't clear. On bursty, mixed-length traffic, which is all real traffic, static batching wastes a lot of the card. ## The fix: batch at the token, not the request **Continuous batching** (also called in-flight batching) drops the idea of a fixed batch entirely. The insight: decode already runs one token-step at a time for every sequence, so treat *each step* as the unit, and let the batch's membership change every step. - When a sequence emits its end-of-text token, it **leaves** the batch immediately, on that step, freeing its slot. No waiting for neighbours. - A waiting request **joins** the batch as soon as a slot opens (after a quick [prefill](/blog/prefill-vs-decode-roofline/) of its prompt), mid-flight, without draining anything. The batch becomes a living set: sequences flow in and out continuously, and the GPU stays as full as there's work to fill it, step after step. A three-word answer is gone in three steps and its slot is instantly reused; the three-thousand-token answer never blocks anyone. That's the whole idea, and it routinely multiplies real-world throughput several times over versus static batching on the same hardware, purely by not wasting slots.

Static batching

Continuous batching

## What it costs, and where it meets its limit Continuous batching isn't free engineering. Sequences entering and leaving every step means the scheduler is juggling the [KV cache](/blog/kv-cache-and-long-context/) for a constantly-changing set of sequences, each of different length. That's precisely why modern servers pair it with paged KV management (storing each sequence's cache in fixed-size blocks, like virtual memory, so slots can be allocated and freed without fragmenting the card). The two ideas go together: continuous batching keeps the compute full, paged KV keeps the memory from becoming the thing that stops you. And the ceiling is, as ever, the [KV cache](/blog/kv-cache-and-long-context/), not the maths. Every sequence in the batch needs its own cache, and on a [single consumer card](/blog/what-the-5090-lacks-vs-datacenter/) with limited memory left after the weights, that cache is what caps how many users you can hold at once. Throughput wants a bigger batch; memory says no. Tuning that trade, batch size against context length against free VRAM, is most of the art of serving an LLM on one card, and it's where the [300 tok/s number](/blog/serving-30b-models-rtx-5090/) is really won or lost the moment more than one person is asking.