What Causes Latency in LLM Applications and How Do You Fix It? (ANSWERED)
Scenario question on LLM latency — prefill vs decode, retrieval overhead, streaming, caching, and model routing fixes.
TL;DR — Quick Answer
LLM latency = retrieval/embeddings + queueing + prefill (processing prompt) + decode (generating tokens sequentially) + post-processing/guardrails. Time-to-first-token (TTFT) dominated by prefill and cold starts; total time by output length. Fixes: streaming UX, smaller/faster models for simple queries, prompt caching/KV cache reuse, speculative decoding, parallel retrieval, reduce output tokens (JSON schema), edge caching of embeddings, and geographic routing to nearest region.
The Interview Question
What causes latency in LLM applications end-to-end, and how would you optimize time-to-first-token and total response time?
Deep Explanation
Latency breakdown
Sign in to unlock full answer
Get deep explanations, PDF export & all LLMs questions
- 7 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions
LatencyStreamingKV CachePerformanceProductionOpenAIAmazonMicrosoft