Deep explanation
Evaluate Anthropic quality in an AI search product
Junior evaluation question: when dashboards look fine but users complain about Claude-generated search answers, build a labeled offline set, check citations and retrieval separately, and add online signals like reformulations and feedback.
Quick answer
Stable metrics with unhappy users usually means we are measuring the wrong thing. I would collect the complaints, label them by failure type, and build an offline test set that checks whether each answer is correct, supported by the cited results, and actually answers the query. Online, I would watch signals like query reformulation, clicks on results after reading the answer, and thumbs-down reasons, and A/B test any prompt or model change against them.
The interview question
Your AI search product uses an Anthropic Claude model to write a short answer with citations on top of the search results. Your dashboard metrics (latency, error rate, an automated quality score) look stable, but users keep complaining that answers are wrong or unhelpful. How would you design offline and online evaluation to find and fix the real quality problem?
Deep explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 27 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions