Chapters 3 & 4

Hardware & Software

From GPU silicon to inference engines. Quick Reference for specs.

Inference runs on datacenter GPUs — cloud (AWS, GCP, Coreweave), on-premise, or air-gapped. NVIDIA leads the market. Three compute types: CUDA Cores (scalar ops — general-purpose), Tensor Cores (matrix multiply — key for inference; GEMM dominates), SFUs (special function units for softmax, GeLU). Measure: FLOPS (FP8/FP4 on newer GPUs), VRAM (HBM3/3e/4, 80–288 GB), bandwidth (TB/s — often the real bottleneck). Peak FLOPS vs sustained: memory-bound workloads hit bandwidth long before compute.

GPU generations

Hopper — H100, H200 (141 GB). FP8, Transformer Engine. Current workhorse for inference. Widely available; mature tooling. Use for stable prod.

Blackwell — B200 (192 GB), B300 (288 GB). FP4, MXFP8, NVFP4, microscaling. Best for new deployments. Higher density, better for large models. GB200 connects two B200s via NVLink to a Grace CPU for unified memory.

Rubin — 2026. HBM4, Vera CPU. Next generation; expect better memory capacity and bandwidth.

Multi-GPU & MIG

NVLink connects GPUs for tensor/expert parallelism — required for models that don't fit on one GPU. MIG (Multi-Instance GPU) splits H100/B200 into up to 7 fractional instances — each gets a slice of compute, VRAM, CPU. For small models (e.g. 3B TTS, embeddings), 2–3 MIG slices can be more efficient than a full GPU: you pay for what you use. MIG has some overhead; measure before committing.

KV cache storage (G1–G4)

When VRAM overflows, KV cache tiers: G1 GPU VRAM (TB/s), G2 CPU RAM (10–100 GB/s), G3 local SSD, G4 networked SSD. Keep hot blocks in G1; migrate cold blocks to G2–G4. GB200 has fast CPU–GPU links for G2 offload. Latency grows as you go down; design for locality. NVIDIA Dynamo with KVBM manages tier migration.

Beyond NVIDIA

AMD MI350, AWS Inferentia/Trainium, Cerebras WSE-3 (wafer-scale, high bandwidth), Groq LPU (SRAM-heavy, very low latency), Google TPU, Qualcomm Cloud AI 100. Each bets on bandwidth, power efficiency, or platform integration. Software and distribution remain challenges — NVIDIA has the ecosystem. Consider alternatives for specific workloads (e.g. Groq for low-latency decode).

Next: Models →← Techniques