Deep explanation
Evaluate Meta Llama quality in an AI search product
Evaluate a Meta Llama answer model in AI search after a migration: quantization effects, chat templates, long-context and multilingual slices, judge calibration, and online A/B design.
Quick answer
I would treat it as a migration evaluation: compare the old model and the new Llama setup side by side on the same queries, sliced by query type, language, and context length, instead of trusting one aggregate score. Llama-specific suspects are quantization lowering quality on hard queries, a wrong chat template, long retrieved contexts, and languages outside Llama's officially supported set. Offline I would use pairwise judging calibrated with human labels, and online an A/B test with behavioural metrics like reformulation and abandonment, deciding per slice where Llama is good enough.
The interview question
Your AI search product recently moved answer generation to a self-hosted Meta Llama model (a quantized Llama 70B-class model behind vLLM) to cut cost. Offline scores and dashboards look stable, but users complain that answers got worse. How would you design offline and online evaluation to find out what changed and decide whether the Llama setup is good enough?
Deep explanation
Simple explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 18 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions