Deep explanation
Evaluate DeepSeek quality in an AI search product
Design offline and online evaluation for a DeepSeek-powered AI search product: hidden failure slices, groundedness and citation checks, reasoning-model latency, language consistency, judge bias, and online experiments.
Quick answer
Stable aggregate metrics usually hide failures in specific slices, so I would start by reading and labelling the complaints to find patterns, such as outdated answers, wrong citations, mixed languages, or slow reasoning queries. Offline, I would build a graded test set per slice and score retrieval relevance, faithfulness to sources, citation accuracy, and freshness, using a calibrated LLM judge plus human labels. Online, I would add behaviour signals like query reformulation and copy events, and run A/B or interleaving tests for each model or prompt change.
The interview question
Your AI search product uses DeepSeek (V3 for answer generation, R1 for complex multi-step queries) to write answers over retrieved web and document results. Dashboards look stable (click-through, latency, thumbs-up rate), but users keep complaining about quality. How would you design offline and online evaluation that actually explains the complaints?
Deep explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 20 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions