RAG Production Interview Guide: Architecture, Failure Modes, and Trade-offs

Production RAG interviews rarely open with textbook definitions. Interviewers want to see whether you can reason about ingestion pipelines, index freshness, access control, and grounding failures when a legal or support assistant serves thousands of tenants with different document corpora. Strong candidates describe the full request path: query normalization, retrieval routing, context assembly, generation, citation verification, and post-generation guardrails.

The most common failure mode in live RAG systems is not a broken embedding model—it is stale or mis-scoped retrieval. Documents get re-ingested without version tracking, chunk boundaries shift silently, and metadata filters exclude the one paragraph that would have grounded the answer. Interviewers probe whether you would instrument recall@k by tenant, track index lag, and separate retrieval misses from generation hallucinations before blaming the LLM.

Latency and cost trade-offs dominate senior RAG design discussions. Hybrid retrieval, reranking, and multi-hop retrieval improve quality but blow through p95 budgets. Candidates who shine explain when to precompute embeddings, when to cache rewritten queries, and how to degrade gracefully—returning a scoped refusal or a retrieval-only summary instead of hallucinating under timeout pressure.

Multi-tenancy adds another layer: ACL enforcement must happen before retrieval, not after generation. Interviewers often present scenarios where answers leak across customer boundaries or cite documents the user cannot access. Your answer should cover metadata-filtered vector search, document-level permissions, and audit trails that tie every citation back to a source version.

Evaluation in production RAG is slice-based, not aggregate. Faithfulness on finance queries may be excellent while HR policy questions fail after a policy PDF update. Discuss golden sets per domain, online citation-click signals, and regression gates that block deploys when groundedness drops on high-risk slices.

Finally, articulate an operational playbook: how you would roll out a new chunking strategy, run shadow indexes, compare offline metrics against live canary traffic, and roll back when NDCG improves but user-reported hallucinations spike. That operational maturity is what separates production engineers from demo builders.