Quick Reference

Cheat sheet for inference engineers. Glossary for full definitions.

Key Metrics

Metric	What it measures	Bottleneck	Typical target
TTFT	Time to first token (prefill latency)	Prefill (compute)	<500ms interactive
TPS / TPOT	Tokens per second (or ms per token)	Decode (memory)	20–60 TPS typical
p50, p95, p99	Latency percentiles	—	p99 <2× p50
Utilisation	GPU compute and memory usage	—	High for batch, varies for interactive

Baseline before optimising. Compare same sequence length, batch size, and hardware.

Hopper — H100, H200 (141 GB). FP8, Transformer Engine. Current workhorse; mature.

Blackwell — B200 (192 GB), B300 (288 GB). FP4, MXFP8, NVFP4. Best for new deployments.

Rubin — 2026. HBM4, Vera CPU. Next generation.

~1B params ≈ 1 GB in FP8. KV cache often 80%+ of remaining VRAM.

Quantization → Need 30–50% speedup without new hardware. FP8 for quality/speed balance. FP4 for max throughput; validate quality.

Speculative decoding → Low batch sizes, spare compute. EAGLE or n-gram for code. Disabled by engines at high batch.

Prefix caching → Code completion, multi-turn chat, agents, RAG with shared context. Put novel content late in the prompt.

Disaggregation → Hundreds of GPUs, prefill/decode have different load patterns. Pair with KV-aware routing.

Tensor Parallelism → Single-node, low latency. Expert Parallelism → MoE, throughput. Context Parallelism → video, long context.

MIG → Small models (<7B) on H100/B200. 2–3 slices often enough for embeddings, TTS, small LLMs.

Chunked prefill → Very long inputs (128K+). Overlap prefill with decode.

Application — OpenAI SDK, LangChain, Vercel AI SDK, LiteLLM

Retrieval — pgvector, Pinecone, RAG pipelines

Serving — vLLM, SGLang, TGI

Engine — TensorRT-LLM, FlashAttention, CUTLASS

Framework — PyTorch, ONNX

Hardware — HBM GPUs, CUDA

Optional: NVIDIA Dynamo for orchestration above engines.

FP16/BF16 — Native. Start here. Baseline for quality checks.

FP8/MXFP8 — 30–50% speedup. Sweet spot. MXFP8 on Blackwell.

FP4/NVFP4 — Max speed, quality tradeoff. Blackwell native. Validate perplexity + evals.

INT8/INT4 (GGUF) — Weights-only; local inference. Q4_K_M, Q5_K_S common.

Production standard: zero perceptible quality loss. Run evals before shipping.