Chapters 2 & 6
Models & Modalities
Architecture, attention, and inference across modalities. Techniques apply to all.
Architecture
LLMs are causal language models. Parse Qwen3MoeForCausalLM: Qwen (family), 3 (version), MoE, CausalLM (decoder-only). Architecture determines runtime support — engines must know layer shapes, attention patterns, and optional components. GQA shares K/V across heads to shrink KV cache.
Transformer blocks: embedding (tokens → vectors), attention, feed-forward (MLP), norm. Linear layers hold most weights; attention is most complex. Output layer (LM head) converts hidden states to logits over vocab. Depth (layers) and width (hidden dim) scale compute and memory. Residual connections and norm placement vary (Pre-LN, Post-LN).
Attention: Q, K, V. Scaled dot-product. Self-attention + causal mask (no lookahead). Multi-head runs parallel heads. KV cache at inference tames quadratic cost — prefill pays O(N²), decode pays O(N) per token. Grouped-query attention (GQA) shares K/V across heads to shrink cache.
MoE: Hundreds of experts (smaller matrices), router activates a subset per token. Qwen3-235B: 22B active of 235B. At batch scale, different requests activate different experts — expect most params active across the batch. Expert Parallelism partitions experts across GPUs. Sparse activation helps latency (fewer ops per token) but routing overhead grows. Good engine support is critical.
Image generation: text encoder (CLIP/LLM) → denoising model (iterative diffusion) → VAE (latent → pixel). LoRAs, ControlNets, refiner. Iterative: 20–50 steps per image. Video adds temporal attention across frames; context parallelism for long videos.
Test your understanding
Why do activation functions like ReLU matter for multi-layer neural networks?