Deep explanation
Evaluate Hugging Face quality in an AI search product
Mid-level evaluation question: build product-specific evaluation for a Hugging Face search stack with labeled query sets, retrieval metrics via sentence-transformers evaluators, reranker and answer checks, versioned datasets, and online signals.
Quick answer
Public leaderboard scores measure other people's data, so first I'd build a labeled test set from our own search logs and complaints. Offline I'd measure each stage separately: retrieval recall and NDCG for the embedding model, how much the reranker improves the top results, and whether the LLM's answer is correct and grounded in the top passages. Online I'd track reformulations, clicks, and thumbs-down by query type, and A/B test model changes before switching.
The interview question
Your AI search product is built on Hugging Face models: a sentence-transformers embedding model for retrieval, a cross-encoder reranker, and an open-weights LLM that writes a short answer. The team chose models from public leaderboards, and your offline metrics look stable, but users still complain about bad results. How would you design offline and online evaluation that reflects your users?
Deep explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 22 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions