Interview ready
Interview Prep
Common questions, tradeoff tables, system design, and numbers to know. Practice before you go in.
Horizontal scaling (more replicas) is the right move for load increase. Keep batch size and latency targets. Quantization and speculation optimise each replica; they don't create new capacity. If TTFT is too high, prefix caching targets prefill directly.
Back-of-the-envelope
| What | Rule / Number |
|---|---|
| VRAM (weights) | ~1B params ≈ 1 GB in FP8; ~2 GB in FP16 |
| KV cache | Often 80%+ of remaining VRAM at long context |
| 70B FP8 | ~70 GB weights; needs 80GB GPU (e.g. H100) + KV headroom |
| TPS per GPU | 20–60 typical for 7B–70B; batch size and length matter |
| TTFT target | <500 ms for interactive UX |
| Quantization gain | FP16→FP8: 30–50% speedup, minimal quality loss |
| Cold start | GPU provision: minutes; model load: 10–60s; compile: cache to cut |
| DeepSeek-V3.1 | 671B; 8× B200 (720 GB) for weights in FP8 |
When to use X vs Y
| Scenario | Choose | Why |
|---|---|---|
| Need speed, no new hardware | Quantization (FP8) | 30–50% gain, quality-safe. FP4 if you validate. |
| Low batch, spare compute | Speculative decoding | Uses idle GPU; 2–3× TPS possible. N-gram for code. |
| Code completion, chat, RAG | Prefix caching | Shared prefixes skip prefill. Put novel content late. |
| Hundreds of GPUs, bursty | Disaggregation | Prefill/decode scale independently; KV-aware routing. |
| Single-node, low latency | Tensor Parallelism | Best per-request latency. All-reduce sync. |
| MoE, throughput focus | Expert Parallelism | Partition experts across GPUs; high throughput. |
| Small model (<7B) | MIG slices | 2–3 slices on H100/B200 for embeddings, TTS. |
Common questions
System design: structure your answer
- 1. Clarify — RPS, latency target, model size, batch vs interactive.
- 2. Estimate — GPUs needed (TPS per GPU, batch size, throughput).
- 3. Components — LB → engine replicas → GPU nodes. Autoscaling.
- 4. Optimisations — Quantization, batching, caching, disaggregation if scale warrants.
- 5. Tradeoffs — Latency vs throughput, cost vs quality, cold start vs overprovisioning.
Quick practice
?
Traffic doubled. You have 10 replicas. What do you do?
?
TTFT is too high. Which optimisation targets it directly?