LLM Inference Optimization: Batching, KV Cache, and Throughput Engineering

LLM inference optimization interviews are where ML meets systems engineering. Interviewers ask you to reason about prefill versus decode bottlenecks, KV cache memory growth with concurrent sessions, and why continuous batching improves GPU utilization but complicates tail latency.

Quantization is not free quality. Discuss INT8/INT4 trade-offs per model family, calibration datasets, and regression eval after quantization. Production teams measure not just throughput but error rate on structured extraction and math-heavy tasks.

Speculative decoding and draft models reduce latency for certain workloads but add routing complexity. Candidates explain when speculation helps chat workloads versus hurts batch embedding jobs, and how to fall back when acceptance rates drop.

Autoscaling LLM serving differs from stateless APIs. Warm pools, model replica pinning, queue depth alerts, and request prioritization tiers all affect user experience. Interviewers present traffic spikes and ask how you protect p99 without bankrupting the GPU budget.

Observability includes tokens per second, time-to-first-token, inter-token latency, batch size distributions, and GPU memory pressure. Debugging a slow endpoint requires attributing time across queueing, prefill, decode, and detokenization—not guessing.