Model Selection failure in an enterprise RAG assistant: how would you respond?

Quick answer

Users say answers are less grounded. Treat that as a Model Selection + RAG contract problem, not a vague quality complaint. Confirm with sliced groundedness (citation coverage, unsupported claims, retrieval hit rate) by route and selected model. Freeze router config, allowlists, prompt/tool versions, and index/embedding versions that shipped with the regression. Pin the previous model map on a canary while keeping retrieval fixed so you can prove whether the new model (or fallback) dropped grounding. Separate selection from confounders: empty retrieval, top-k changes, reranker off, truncated context, citation post-process bugs. Mitigate with pin/rollback of the router, force grounded models for high-risk asks, and require citation checks before answer. Ship durable gates: model-route golden set for groundedness, per-model cost/latency budgets, and canary rollback when groundedness falls.