Transformer Architecture(8)
What does scaled dot-product attention compute?
Why divide dot products by √d_k in attention?
Time and space complexity of standard self-attention w.r.t. sequence length n?
Total parameter count for all projection matrices in one MHA layer (d_model dims)?
Key difference between GPT (decoder-only) and BERT (encoder-only) attention?
What is Rotary Positional Embedding (RoPE)?
What is the purpose of the Feed-Forward Network (FFN) sublayer in a transformer block?
In most LLMs, the FFN hidden dimension is set to:
Tokenization(4)
How does Byte Pair Encoding (BPE) build its vocabulary?
A model has vocab size 100k and embedding dim 4096. How large is the embedding matrix?
What is 'tokenization fertility' and why does it matter?
Why do LLMs struggle with character-level tasks like reversing strings?
Training(6)
What is the training objective of a standard autoregressive LLM?
What does 'teacher forcing' mean during LLM training?
What is gradient checkpointing and what does it trade off?
What is the Chinchilla scaling law's key insight?
What is mixed-precision training (BF16/FP16)?
Why is BF16 often preferred over FP16 for LLM training?
Inference & Decoding(8)
What are the two phases of LLM inference and how do they differ?
What is the KV Cache and why is it critical for inference?
How does KV cache size scale with sequence length?
What is continuous batching (iteration-level scheduling)?
What problem does PagedAttention (vLLM) solve?
What is speculative decoding?
What is the key bottleneck during the decode phase?
What is 'arithmetic intensity' and why does it matter for LLM inference?
Quantization(5)
What does INT8 quantization do to model weights?
What is the difference between Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT)?
What is GPTQ?
What does AWQ (Activation-Aware Weight Quantization) do differently from GPTQ?
What is the key challenge with quantizing activations compared to weights?
Parallelism(4)
What is Tensor Parallelism (TP)?
What is Pipeline Parallelism (PP)?
What is the 'pipeline bubble' problem in pipeline parallelism?
What is Data Parallelism and how does it differ from Model Parallelism?
Fine-tuning(5)
What does LoRA (Low-Rank Adaptation) do?
What is the rank `r` in LoRA, and what effect does increasing it have?
What is RLHF and what are its three stages?
What is DPO (Direct Preference Optimization) and how does it simplify RLHF?
What is 'catastrophic forgetting' in fine-tuning and how does LoRA help?
Sampling(4)
What does temperature do in LLM sampling?
What is Top-P (nucleus) sampling?
What is the difference between greedy decoding and beam search?
What causes 'repetition' in LLM outputs and how does repetition penalty address it?
Context & Memory(4)
What is the 'lost in the middle' problem in long-context LLMs?
What is RoPE scaling / YaRN and why is it needed?
What is sliding window attention?
What is Flash Attention and what does it optimize?
Evaluation(4)
What does perplexity measure in a language model?
What is TTFT and TPOT, and why do they matter for LLM serving?
What is 'tokens per second per GPU' and what does it reveal?
What is the MFU (Model FLOP Utilization) metric?