Supervised ranker failure in an enterprise RAG assistant: how would you respond?

Question

You are the junior engineer on call for an enterprise RAG assistant. Retrieved chunks are still returned, but answers have become less grounded in the source documents. The assistant uses a supervised model, trained on labeled query-passage pairs, to choose which retrieved chunks the generator may use. How would you investigate and fix that without breaking the SLA or the budget?

Accepted Answer

I would not retrain the generator or rebuild the index first. I would check whether the supervised chunk selector is rejecting the passages that actually support the answer. I would compare a sample of bad queries with the labeled validation set, look for a shift in document type or in how "good passage" was defined, and only then relabel and retrain that small model if production no longer matches those labels.

Supervised ranker failure in an enterprise RAG assistant: how would you respond?

The interview question

Deep explanation

1. Short Interview Answer