Auxiliary Loss. The incorporation of auxiliary loss [23, 56] helps mitigate the risk of overfitting by
promoting the diversification of the experts’ knowledge and improving the model’s generalization
capabilities for sparsely gated mixture-of-expert models. Furthermore, auxiliary losses can be
employed to address specific issues, such as load balancing among experts or preventing expert
collapse, which can further enhance the model’s overall performance. We experiment with both
balancing loss that is used in [23] and router Z-loss that is used in [56] in Table 2. The implementation
of balancing loss contributed to enhanced performance on MMLU, BBH, and GSM8K for FLAN-

7
Mixture-of-Experts