Evaluate Scale Challenges quality in an AI search product

Quick answer

When aggregate metrics look fine but users still complain about an AI search product, treat it as a scale-and-slice problem: overall averages hide bad cohorts. Design offline eval with a stratified golden set (head queries, long-tail, new docs, multilingual, adversarial) scored on relevance, groundedness, and citation quality—not only click-through. Pair that with online eval: interleaved or A/B tests plus slice dashboards for latency, zero-result rate, and complaint rate by query class. Ship changes behind canaries with explicit rollback when any critical slice worsens even if the global score holds.