Mixture of Experts (MoE) Architecture Interview Question (EXPLAINED)
Hard model question on MoE — routed FFN experts, sparse compute, capacity factors, and Mixtral/GPT-4 class rumors.
TL;DR — Quick Answer
MoE replaces dense FFN layers with multiple expert networks; a router selects top-k experts per token (sparse activation). Total parameters grow while **activated** parameters per token stay smaller — compute-efficient scale. Training needs load balancing losses to avoid expert collapse. Inference challenges: memory for all experts on GPU, routing overhead, batching uneven experts. Examples: Mixtral 8x7B, DBRX, rumored frontier MoEs — interviewers test whether you confuse total params with active params.
The Interview Question
Explain Mixture of Experts (MoE) in LLMs. How does sparse activation work and what are training and inference implications?
Deep Explanation
Dense vs MoE
Sign in to unlock full answer
Get deep explanations, PDF export & all LLMs questions
- 11 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions
MoEMixture of ExpertsArchitectureRoutingScaleMetaMistralGoogle