Deep explanation
Evaluate Cost Reduction quality in an AI search product
Evaluate cost-reduction changes in AI search: per-change ablations, slice analysis, semantic cache correctness, pairwise judging, online A/B with behavioural metrics, and cost per successful query.
Quick answer
I would stop treating the cost project as one change and evaluate each lever separately: the smaller model, fewer passages, shorter prompt, and semantic cache. Offline, I would rebuild the test set from real and complained-about queries, slice it by query type, and run ablations so each lever's quality impact is visible. Online, I would A/B test the levers with behaviour signals like reformulation and abandonment and judge success by cost per successful query, keeping only the savings that do not hurt users.
The interview question
You led a cost-reduction project on an AI search product: a smaller answer model, fewer retrieved passages, shorter prompts, and a semantic cache. Cost per query dropped and offline scores look stable, but users now complain about answer quality. How would you design offline and online evaluation to prove where the cost cuts hurt quality and which ones are safe to keep?
Deep explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 21 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions