Chapter 0
Inference
Inference is the second phase in a generative AI model's lifecycle: Training learns model weights from data. Inference serves generative AI models in production. In the last decade's ML boom, many data scientists knew both. Classic ML inference (e.g. XGBoost on CPUs) was straightforward. Generative AI inference is not — you can't simply take weights, get GPUs, and expect fast, reliable production. It requires three layers working together.
- Runtime: Optimising a single model on a single GPU-backed instance. CUDA → PyTorch → inference engines (vLLM, SGLang, TensorRT-LLM). Kernels like FlashAttention matter.
- Infrastructure: Scaling across clusters, regions, clouds without silos. At first: autoscaling. Past a few hundred GPUs: capacity, multi-region, multi-cloud. Ultimately: a global unified pool of compute.
- Tooling: The right abstraction for engineers — control with productivity. Not black-box API, not raw compute.
The runtime layer ensures each model runs as performantly as possible. It depends on CUDA, PyTorch, inference engines. Model performance techniques apply across modalities:
- Batching — Run requests in parallel, weave token-by-token.
- Caching — Reuse KV cache between requests that share prefixes.
- Quantization — Lower precision to access more compute and reduce memory.
- Speculation — Generate and validate draft tokens for multiple tokens per pass.
- Parallelism — Efficiently leverage multiple GPUs.
- Disaggregation — Separate prefill and decode onto independently scaling workers.
These apply to VLMs, embeddings, ASR, TTS, image and video generation — not just LLMs. Runtime optimisations alone are not enough. With enough traffic, a single instance is overwhelmed. That's an infrastructure problem: autoscaling, capacity, multi-region, bin-packing across clouds, unified GPU pools.
Which layer is responsible for knowing when to add and remove replicas?