Deep explanation
Evaluate MLflow quality in an AI search product
Senior evaluation question: fix a misleading MLflow evaluation setup for AI search using production traces, human feedback, calibrated LLM judges, retrieval metrics, versioned evaluation datasets, and release gates.
Quick answer
Stable scores with rising complaints means the evaluation set or the judges no longer match real traffic. I'd pull complaint traces from MLflow, label them, and turn them into a versioned evaluation dataset, then run separate retrieval and answer scorers with mlflow.genai.evaluate and check each judge against human labels. Online, I'd attach user feedback to traces and compare scores by query segment, and only promote a new model or prompt alias when it improves on the refreshed set.
The interview question
Your AI search product (retrieval plus an LLM-written answer) is tracked in MLflow: each release logs evaluation runs with LLM-judge scores, and production requests are traced. The judge scores have been stable for months, yet user complaints are rising. How would you use MLflow to redesign offline and online evaluation so the scores actually reflect what users experience?
Deep explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 26 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions