AI Evaluation Pipelines: Offline Gates, Online Signals, and Regression Discipline

AI evaluation pipeline interviews distinguish teams that ship confidently from teams that pray on deploy day. Interviewers ask how you structure offline evals, online monitoring, and human review loops so that prompt, model, and retrieval changes are measurable before they hit users.

Golden datasets decay. Candidates explain versioning, stratified sampling, adversarial case injection, and ownership models so eval sets stay representative as products evolve. A golden set frozen in 2024 will not catch 2026 failure modes.

LLM-as-a-judge scales eval but needs calibration. Discuss rubric design, pairwise versus pointwise scoring, inter-rater agreement with humans, and detecting judge drift when the judge model itself updates. Ensemble judges and anchor examples reduce variance.

CI/CD integration means eval gates with clear pass-fail criteria per slice—not one global threshold. Interviewers want examples: block deploy if compliance slice faithfulness drops 2%, even if marketing slice improves. Shadow traffic and canary evals bridge offline to online.

Online evaluation closes the loop with implicit signals—thumbs, re-queries, escalation tags—and explicit human audits. Articulate how you avoid optimizing for clickbait answers and how you detect metric gaming when teams pressure eval numbers before launches.