Debug Model Selection regression in a multi-tenant AI support platform

Quick answer

Confirm the symptom is real with sliced metrics: p95 latency up, quality flat, broken down by tenant, route, and selected model. Freeze what shipped in the same deploy—router config, model allowlists, prompt versions, retrieval index, and tool schemas—then pin versions so you can A/B against the previous router. Separate model selection from confounders like cache misses, cold starts, retrieval fan-out, tool timeouts, and queue depth. Compare traffic that landed on model A versus model B before and after the release, using canary or shadow tenants if available. Typical selection regressions: the router flipped traffic to a slower larger model, a fallback storm hit the expensive path, a per-tenant override broke, or a cost/latency budget mis-fire. Mitigate by pinning the previous router config, rolling back canaries, and adding joint latency-plus-quality gates on every router change.