Mixture of Experts (MoE) Interview Guide
Explain sparse expert routing, capacity, load balancing, communication, and serving trade-offs in modern MoE models.
Core idea
A sparse MoE layer routes each token to a small subset of experts rather than executing every expert. This can increase parameter capacity without proportional per-token compute, but routing and communication become first-class systems problems.
Routing and balance
Discuss top-k expert selection, router confidence, capacity limits, dropped or rerouted tokens, auxiliary balance objectives, and the risk that a small set of experts becomes overloaded.
Distributed serving
Expert parallelism places different experts on different accelerators. Performance depends on token-to-expert skew, all-to-all communication, locality, batching, and how expert parallelism interacts with data/tensor parallelism.
Interview trade-offs
Compare dense vs MoE on quality-per-FLOP, memory footprint, communication, utilization, fault tolerance, and deployment complexity. Measure tail latency and expert imbalance, not only throughput.