Deep explanation
Production incident: RAG Quality Improvement outage in a compliance review automation system
Behavioral and incident question: a RAG quality release (reranker, higher top-k, longer context) causes failures and doubled cost at peak in a compliance system. First 30 minutes, recovery, and lessons.
Quick answer
My first move is to protect the compliance queue so no case is marked reviewed by mistake, then roll back the release behind its feature flag, because the timing points straight at it. The likely mechanism is that the reranker added latency under load, requests timed out and were retried, and the bigger top-k and longer passages roughly doubled tokens per request, which together explain the failures and the cost. After recovery I would replay stuck cases, then reintroduce the improvement gradually with load tests, latency budgets, and token caps.
The interview question
You shipped a RAG quality improvement to a compliance review automation system: a cross-encoder reranker, a higher retrieval top-k, and longer context passages. At peak traffic the next day, requests start failing, LLM cost doubles, and compliance reviewers open tickets because cases are stuck. Describe your first 30 minutes and your recovery plan, and what you would say about this incident in an interview.
Deep explanation
Simple explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 19 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions