Chapter 5
Techniques
New research goes live in production within months. The more constraints you introduce, the better performance you'll achieve. Expand sections below for details.
KV-aware routing, disaggregation, and dynamic model parallelism only make sense with a large number of GPUs. Tuning isn't one-time — iterate via deployments or runtime adjustments. One Baseten engineer tried 77 configurations before doubling TPS for a customer's autocomplete model. Quantization tip: Linear weights are least sensitive; attention is most sensitive.
Techniques from research reach production in months. The more constraints you introduce, the better performance — but techniques can be symbiotic or incompatible. Quantizing KV cache helps disaggregation; large batch sizes reduce speculation headroom. Aim for a balanced set of optimisations.
Check your understanding
Which component is LEAST sensitive to quantization?
Why do engines disable speculative decoding at high batch sizes?