Offline vs Online Evaluation for GenAI Products (ANSWERED)
Scenario question on offline golden eval vs online A/B, feedback, and guardrail metrics in production.
TL;DR — Quick Answer
Offline eval runs fixed golden sets before deploy — faithfulness, accuracy, rubric scores, regression gates, reproducible. Online eval measures live behavior — thumbs down, task completion, escalation rate, latency SLO, safety incident rate, A/B experiments. Offline catches regressions cheaply; online catches distribution shift and UX issues golden sets miss. Use both: offline blocks bad releases; online validates real users; connect via shadow mode and canary. Never ship on offline alone without online monitoring.
The Interview Question
Compare offline and online evaluation for generative AI products. What metrics belong in each layer?
Deep Explanation
Offline
Sign in to unlock full answer
Get deep explanations, PDF export & all LLMs questions
- 9 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions
EvaluationOfflineOnlineA/B TestingMetricsGoogleMetaStripe