Deep explanation
Transformers failure in an enterprise RAG assistant: how would you respond?
Senior RAG incident question: trace ungrounded answers in a Transformers-based RAG stack to unpinned model revisions, chat template drift, truncation side, generation config, padding, dtype or quantization, and embedding mismatches.
Quick answer
A rebuild with "no model change" can still change the model, tokenizer, chat template, generation defaults, or library behavior if revisions and versions aren't pinned. I'd compare the old and new image on the same inputs: model and tokenizer commit hashes, the exact prompt string after the chat template, how long contexts are truncated, generation settings, dtype or quantization, and whether the embedding model still matches the index. Then I'd pin everything, fix the specific difference, and confirm on a groundedness eval before rolling out.
The interview question
Your enterprise RAG assistant runs a self-hosted open-weights model with Hugging Face Transformers, plus a sentence-transformers embedding model for retrieval. After an image rebuild with no intended model change, answers became less grounded: they ignore retrieved passages or invent details. As the senior engineer on call, how would you investigate Transformers-specific causes and fix it without breaking the latency SLA or GPU budget?
Deep explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 23 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions