Prefill & decode
Every LLM request has two phases. Different bottlenecks → different optimisations.
Prefill processes the entire prompt in parallel and produces the first output token — it determines TTFT. Decode generates subsequent tokens one by one and drives TPS. Different bottlenecks require different optimisations.
Every LLM request goes through exactly two phases. Confusing them is a common mistake when optimising for latency. Each phase has different bottlenecks and therefore different optimisation strategies. The inference loop: tokeniser converts text to tokens → prefill builds KV cache → decode generates tokens one by one until stop token or max length.
Phase 1 — Prefill
The model ingests the entire prompt in parallel. All input tokens are processed at once through every layer. This phase is compute-bound and relatively fast. It determines Time to First Token (TTFT) — the latency users perceive before any output appears.
During prefill, the model weights are loaded once, then a series of large matrix multiplications occurs between the input matrix and the attention matrices. This is a lot of calculations versus a single read from memory, creating high arithmetic intensity. The phase populates the KV cache: for each input token, the attention mechanism computes keys and values that are stored for reuse. Without the KV cache, every decode step would need to recompute attention over the entire sequence — unbearably slow.
At the end of prefill, the model produces a vector of logits (scores for each token in the vocabulary). Sampling (greedy, temperature, top-k, top-p) selects the first output token. The loop then enters decode.
Phase 2 — Decode
The model generates output tokens one at a time. Each step: take the latest token, run it through the model (with the existing KV cache), append new keys/values to the cache, produce logits, sample the next token. This phase is memory-bandwidth-bound — adding more FLOPS rarely helps because the bottleneck is loading weights and KV cache from VRAM. It determines Tokens Per Second (TPS) or Time Per Output Token (TPOT) — perceived streaming speed.
On decode, the model weights are loaded for every token, which is generated via much less expensive vector-matrix multiplication (one new token in, one token out). Relatively few floating-point operations are needed compared to loading the entire model weights, so the arithmetic intensity is low. The KV cache grows with every output token; at long context lengths (32K, 64K, 128K+ tokens) it becomes the dominant consumer of GPU memory. A 4096×4096 attention matrix in FP16 is ~32 MiB — and that's just one head, one layer.
Decode continues until the model emits a stop token or hits the max output length. For structured output (JSON, tool calls), logit biasing and schema constraints apply after each forward pass.
Why this matters for optimisation
- Techniques like speculative decoding target the decode phase — they use spare compute (idle at low batch sizes) to validate draft tokens and produce N+1 tokens per pass.
- Prefill batching has different economics than decode batching; you batch differently for each. Prefill benefits from batching many prompts together; decode benefits from continuous batching that weaves requests token-by-token.
- Disaggregation separates prefill and decode onto different workers because they need different hardware profiles — prefill wants more FLOPS, decode wants more memory bandwidth.
- Prefix caching reuses the KV cache from prefill when requests share a prefix — skipping redundant prefill work.
Test your understanding
Prefill is compute-bound and decode is memory-bound. Why can't we just add more GPUs to speed up decode?
Check your understanding
Which phase determines Time to First Token (TTFT)?
Why is LLM decode memory-bound rather than compute-bound?