RAG evaluation interview prep without memorizing metrics

A quick note first. The candidate in this story is an illustrative composite, not a real person. The prep approach is real advice, but nothing here reports a real company's interview or result.

For a while I prepared for RAG evaluation questions the way I prepared for exams. I made a list of metric names, wrote a one-line definition next to each, and read it before every interview. It did not work. The moment an interviewer asked a follow-up, such as which of these would have caught the bug in my last project, my list was useless. I knew words. I did not know what any of them were for.

What changed was a simple idea. Every RAG evaluation question is really asking one of three things. Did the system find the right context? Did the answer stay faithful to that context? Did the answer actually help the person who asked? If I could place a question into one of those three, I could reason my way to a good answer even if I forgot the name of a metric.

So I stopped memorizing and started building. I took a small question-answering project I had already made and wrote twenty questions by hand. For each one I wrote down which passage should come back and what a good answer would say. That small set became the center of my prep.

The first thing I practiced was evaluating retrieval on its own. For each question, I checked whether the right passage was in the results at all, and how high it ranked. When an answer was wrong, I looked here first. Many of my wrong answers turned out to be retrieval misses, not generation problems. In an interview, being able to say that I check retrieval before blaming the model turned out to be more convincing than any metric name. The RAG debugging walkthrough at /blog/how-to-debug-a-rag-pipeline-retrieval-ranking-and-grounding shows how to split failures this way.

The second thing I practiced was faithfulness. I looked for answers that sounded right but included claims the retrieved passages did not support. Sometimes the model ignored the context completely and answered from memory. The guide at /blog/model-ignores-retrieved-context-rag-interview-guide covers that failure and how to talk about it.

The third thing I practiced was usefulness. Some answers were faithful and still unhelpful, because they missed the actual question or buried the answer in caveats. I learned to check these separately, and to explain why a system can pass one check and fail another.

Then I practiced the follow-up questions that come up around evaluation, using my own project as the example every time. How would you know this is ready to ship? What would you run on every change? How would you use a model as a judge, and how would you check that judge? What do you do when two checks disagree? The LLM evaluation guide at /blog/llm-evaluation-interview-guide and the RAG interview guide at /blog/rag-interview-guide were the two I returned to most.

The last piece was telling the story out loud. I practiced a two-minute answer: my test set, the three checks, one failure I found, and what I changed because of it. I recorded myself and cut every sentence that named a metric without saying what it caught.

I still know the common metric names. But in interviews I lead with the question each one answers and the failure it would catch in a real system. If you want to see how the opposite approach goes wrong, the composite story at /blog/failed-first-rag-deep-dive shows what happens when evaluation is the last thing you think about.