Amazon Interview: Cost-Aware LLM Routing (EXPLAINED)
Hard Amazon company question — model cascades, unit economics, SageMaker/Bedrock routing, and SLO-aware cost optimization.
TL;DR — Quick Answer
Router classifies query difficulty/intent using cheap model or rules; cascade to mid-tier then frontier only when needed; track cost per successful task; cache embeddings and semantic answers; batch offline workloads; autoscale inference endpoints; use Bedrock/SageMaker multi-model endpoints; enforce tenant budgets; SLA tier maps to model class (premium vs standard). Measure quality regression when routing aggressively — golden set gates routing policy updates. Observability: cost/query, escalation rate, p95 latency by tier.
The Interview Question
Amazon-style interview: Design cost-aware routing for an LLM platform serving heterogeneous workloads at AWS scale.
Deep Explanation
Sign in to unlock full answer
Get deep explanations, PDF export & all LLMs questions
- 12 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions