Click any card to flip and reveal the definition. Learn by doing.
Time to First Token. The latency between sending a request and receiving the first output token. Dominated by the prefill phase. Target: <500ms for interactive use.
Tokens Per Second. Inter-token latency during decode. Bottlenecked by memory bandwidth. Typical: 20–60 TPS.
Stores attention keys/values from prefill. Primary VRAM consumer — often 80%+ of memory for long context.
Compute-bound phase. Process input, build KV cache. O(N²) in length. Determines TTFT.
Memory-bound phase. Generate tokens one by one. Each step reads full KV cache. Determines TPS.
Token-level interleaving so GPU slots stay busy. vLLM, SGLang, TensorRT-LLM use it.
Draft tokens → target verifies → N+1 tokens per pass. Best at low batch sizes.
Lower precision (FP16→FP8→FP4) for speed and memory. FP8: 30–50% gain with minimal quality loss.
Separate prefill and decode onto different workers. Scale each independently.
Reuse KV cache for shared prompt prefixes. Major win for code completion, chat, agents.
Fused attention kernels. Essential for prefill and long context. Lossless.
KV in fixed-size pages. Reduces fragmentation. vLLM popularised it.
Mixture of Experts. Hundreds of small matrices, router activates subset per token. Qwen3-235B: 22B active of 235B.
Time from scale-up to first response. GPU provision → image → model → compile. Optimise each.
Multi-Instance GPU. Split H100/B200 into up to 7 fractional instances. Good for small models.
Split tensor ops across GPUs. Best per-user latency. All-reduce sync.
Partition MoE experts across GPUs. High throughput for large MoE models.