Why Your LLM Is Slow for Three Completely Different Reasons

Picture a kitchen with exactly one chef and one recipe book sitting on a shelf across the room. Two things happen in this kitchen, and they are not the same kind of work at all.

When an order comes in, the chef reads it in full, then preps every ingredient in it at once - dicing, measuring, laying everything out on the counter in parallel, because the whole order is already known. That is fast, and it gets faster if you hire more hands.

Plating the dish is different. It happens one step at a time - add the sauce, walk to the shelf, check the recipe, come back, add the next component, walk to the shelf again. Each step depends on the one before it, and each step means another walk to the shelf and back, no matter how skilled the chef is with a knife.

That kitchen is a rough sketch of what happens inside a GPU every time you send a prompt to an LLM and wait for tokens to stream back. And the split between those two chef-tasks explains something that trips up a lot of people building on top of LLMs: why the fixes that make your app faster in one dimension often make it slower in another.

What actually happens between hitting enter and seeing the first word?

An LLM answering a prompt does two distinct jobs, back to back.

The first job is called prefill. Your entire prompt - every token of it - gets fed into the model at once. Because self-attention lets every token look at every other token, and every token in your prompt is already known before processing starts, the model can process all of them in parallel, in one shot. This is the “prep the whole order at once” step.

The second job is decode. The model produces your reply one token at a time, and each new token depends on every token that came before it, including the ones the model itself just generated. There is no shortcut here - token 501 cannot be computed until token 500 exists. This is the “walk to the shelf for every single plating step” job.

Here’s the detail that catches people off guard: these two jobs are not just structurally different, they are bottlenecked by two completely different physical resources on the chip.

Why does reading a prompt strain a different resource than writing the reply?

Every layer of a model holds large matrices of numbers, its weights, sitting in the GPU’s main memory (HBM, the GPU equivalent of RAM). To do any computation, those weights first have to be pulled out of that memory and into the much smaller, much faster area next to the actual arithmetic units, and only then can the multiplication happen. That pull is not free. It takes real time, and it’s a fixed cost regardless of how much you do with the weights once they arrive.

During prefill, once a layer’s weights are sitting in that fast area, the model multiplies them against every token in the prompt at once, maybe hundreds of tokens. One expensive fetch pays for hundreds of multiplications. Most of the GPU’s time goes to actual arithmetic, hence: compute-bound.

During decode, that same fetch happens, the same weights come from the same shelf, but they get multiplied against exactly one token, because that’s all you have at that step. One expensive fetch pays for a single multiplication, and by the time you generate the next token, those weights are gone from the fast area (there wasn’t room to keep the whole model there) and have to be fetched all over again. Most of the GPU’s time goes to waiting for data to arrive, not computing on it, hence: memory-bandwidth-bound.

This is why decode does not get meaningfully faster if you buy a GPU with a faster arithmetic engine but the same memory bandwidth. The chef isn’t slow at chopping. The chef is slow because of the walk to the shelf, and a faster knife does nothing about a walk.

There’s a second piece of that walk worth naming: the KV cache. When prefill processes your prompt, it computes a “key” and “value” vector for every token, at every layer, and these get stored, because every future decode step needs to look back at all of them to know what’s already been said. That storage is the KV cache, and it’s large enough that it has to live in the same slow main memory as the weights, not in the fast area. So every decode step doesn’t just re-fetch the weights, it also re-fetches the growing pile of every previous token’s key and value vectors, at every layer.

That’s why a long prompt costs you twice. A 5,000-token prompt takes longer to prefill, because there’s more to chop up front. And it also makes every single token generated afterward slower, because every decode step now has a bigger cache to read through, at every layer, forever, for the rest of that conversation.

If batching helps, why would it ever make things worse?

The standard fix for decode’s bad math, one expensive weight-fetch paying for one measly multiplication, is batching. If eight different users are all mid-decode at the exact same layer, they all need the identical weights. So fetch those weights once, and multiply them against all eight users’ single tokens at the same time. One fetch, eight multiplications instead of one. Decode starts looking more like prefill’s efficient ratio, and this is why every serving system in production (open-source engines like vLLM and TensorRT-LLM, purpose-built for exactly this) batches decode steps across many users continuously rather than serving one request at a time. Throughput, and cost per token, both improve directly from this.

Here’s where it turns. Picture that batch of eight, quietly decoding together, sharing weight-fetches. A ninth, brand-new request shows up, and it needs prefill, not decode - it needs to dump its whole prompt onto the GPU’s compute engine in one big parallel chunk.

Time  ---->
GPU:  [decode x8][decode x8][ PREFILL (new request) ][decode x8][decode x8]
                              ^
                  the new request's prompt has to land here -
                  either it waits for a slot, or the 8 in-flight
                  users' decode steps wait behind it instead

If the scheduler makes the new request wait for a clean slot, its time-to-first-token gets worse - it’s sitting in a queue while nothing computes for it. If instead the scheduler folds its prefill into the current step so the batch stays full, the other eight requests’ decode steps get delayed, because prefill is compute-hungry and now competes for the same cycles decode was using lightly. Either way, someone pays a latency cost. Batching helps the metric everyone tracks in aggregate (tokens per second, dollars per token) and hurts the metric a single new user actually feels (how long the first word takes to arrive).

That’s the core of it: TTFT and tokens-per-second are opposite pulls, because they come from optimizing for one request at a time versus optimizing for many requests sharing a chip.

Why doesn’t one fix, quantization, just solve one of these?

Weights are normally stored as 16-bit numbers. Store them as 8-bit or 4-bit numbers instead, and the total size of every weight matrix shrinks by half or more. Since decode’s whole problem is how many bytes have to move across a bandwidth-limited pipe, shrinking those bytes directly speeds up decode. Less to carry, same pipe, less time per trip.

You’d expect this to do little for prefill, since prefill was never waiting on the pipe - it was busy on the engine. But there’s a second effect: GPUs have specialized hardware that performs low-precision arithmetic faster than high-precision arithmetic, often close to double the speed for 8-bit versus 16-bit. Prefill’s bottleneck is the engine, and lower precision speeds the engine up too. So quantization helps both jobs, but through two unrelated mechanisms: it shrinks the pipe’s cargo for decode, and it speeds up the engine itself for prefill. The cost, unsurprisingly, is precision. Fewer bits per number means a coarser approximation of the original weights, which is a real quality trade-off, not a free win.

What if the GPU has speed to spare and a request still can’t start?

Everything so far has been about speed - how fast the engine computes, how fast the pipe moves data. There’s a third constraint that has nothing to do with speed at all: space.

The GPU’s main memory is a fixed size. A chunk of it is permanently reserved for the model’s weights and never shrinks no matter how many users you serve. Whatever’s left has to hold the KV cache for every conversation currently in flight, and a conversation with ten thousand tokens of history needs a proportionally bigger slice of that space than one with five hundred.

As more users connect, or existing conversations run longer, that remaining space fills up. And unlike a queue waiting on a busy engine or a busy pipe, there’s no swap-it-out-and-come-back-later option that doesn’t cost a huge, separate latency penalty - roughly the equivalent of walking out to a warehouse instead of the shelf across the room. Once every table in the kitchen is occupied by a dish still being built, a new order genuinely cannot start, not because the chef is slow, but because there is nowhere to put it. The two real options left are to make the new request wait, or to evict an existing conversation’s cache to free the room.

This is why production systems build admission control (refuse or queue new requests once memory crosses a threshold) and eviction policies (drop the least-recently-used conversation’s cache) - entirely separate machinery from anything that speeds up compute or bandwidth. And it’s why quantization shows up here too, for a third, distinct reason: a smaller KV cache per token means more conversations fit into the same fixed space before you hit the wall.

Three bottlenecks, three different repair kits

An LLM answering you is never bottlenecked by one thing called speed. It’s bottlenecked by an engine (compute, the limit on prefill), a pipe (memory bandwidth, the limit on decode and the reason long prompts and KV caches slow down every subsequent token), and a room (memory capacity, the limit that shows up only once you’re serving many conversations at once and has nothing to do with speed at all).

Reach for the wrong repair kit and you can optimize hard and still ship something that feels slower to the person waiting on it. If time-to-first-token is the complaint, look at what’s competing for the engine: prompt length, scheduling, how much a new request has to wait behind other people’s decode steps. If tokens-per-second is the complaint, look at what’s straining the pipe: batch size, quantization, how big the KV cache has grown. And if requests are queueing or getting cut off even when the GPU looks idle, the problem was never speed - it was that the room ran out of tables.

Which of the three feels hardest to fix without a genuine architecture decision, rather than a config change: the engine, the pipe, or the room?

What’s your take? Drop a comment below.