Deep explanation
Debug vLLM regression in a multi-tenant AI support platform
Senior debugging question: isolate a vLLM latency regression using KV cache usage, preemptions, waiting queue depth, prefill vs decode timing, prefix caching, chunked prefill, and per-tenant token load.
Quick answer
Flat quality with worse latency points at scheduling and memory, not the model. I'd read vLLM's Prometheus metrics: waiting queue length, KV cache usage, preemption count, and time to first token versus time per output token, then split traffic by tenant and prompt length. Long prompts from the new tenant can fill the KV cache and cause preemptions or long prefills that block other requests, so the fixes are things like chunked prefill, tuned batch token limits, prefix caching, per-tenant limits, or a separate pool for long-context traffic.
The interview question
Your multi-tenant support platform self-hosts an open-weights model on vLLM. After a release that upgraded vLLM, raised max_model_len, and onboarded a tenant who sends very long ticket histories, p95 time to first token and end-to-end latency spiked while answer quality stayed flat. How would you isolate the cause using vLLM's scheduler and KV cache behavior, and what would you tune?
Deep explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 25 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions