RAG Evaluation and Debugging: Metrics, Slicing, and Root-Cause Isolation

RAG debugging interviews center on structured triage. When users report wrong answers, the first question is whether the retriever returned the right context, whether the generator ignored good context, or whether the evaluation harness itself is stale. Candidates who jump straight to prompt tuning without isolating the retrieval layer typically fail these rounds.

Offline metrics like context precision, context recall, and answer faithfulness only help when your golden set reflects production query distribution. Interviewers expect you to discuss building representative eval slices: long-tail product SKUs, ambiguous acronyms, multi-hop questions, and queries that require metadata filters. A single aggregate F1 score is a red flag in mature teams.

LLM-as-a-judge accelerates iteration but introduces judge bias and position bias. Strong answers explain when to use human rubrics, when to ensemble judges, and how to calibrate automated scores against production signals like citation clicks, thumbs-down reasons, and support escalations. The best candidates treat judges as another model with its own drift problem.

Debugging workflows should be trace-driven. Each failed example needs the rewritten query, retrieved chunks with scores, reranker output, assembled prompt, model route, and final citations. Interviewers often ask how you would build a replay harness to test whether a new embedding model fixes retrieval without re-running full generation.

Regression gates belong in CI/CD for RAG, not just in notebooks. Describe eval gates that block prompt changes, index migrations, and chunking updates when faithfulness drops on compliance-critical slices—even if average benchmark scores improve.