Demons in the Detail: On Implementing Load Balancing Loss for Training
  Specialized Mixture-of-Expert Models
This paper revisits the implementation of
Load-balancing Loss (LBL) when training
Mixture-of-Experts (MoEs) models. Specifically, LBL for MoEs is defined as N_E
sum_{i=1}^{N_E} f_i p_i, where N_E is the total number of experts, f_i
represents the frequency of expert i being selected, and p_i denotes the
average gating score of the expert i. Existing MoE training frameworks
usually employ the parallel training strategy so that f_i and the LBL are
calculated within a micro-batch and then averaged across parallel
groups. In essence, a micro-batch for training billion-scale LLMs normally
contains very few sequences. So, the micro-batch LBL is almost at the sequence
level, and the router is pushed to distribute the token evenly within each
sequence. Under this strict constraint, even tokens from a domain-specific
sequence (e.g., code) are uniformly routed to all experts, thereby
inhibiting expert specialization. In this work, we propose calculating LBL
using a global-batch to loose this constraint. Because a
global-batch contains much more diverse sequences than a micro-batch, which
will encourage load balance at the corpus level. Specifically, we introduce an
extra communication step to synchronize f_i across micro-batches and then use
it to calculate the LBL. Through experiments on training MoEs-based LLMs (up to
42.8B total parameters and 400B tokens), we surprisingly
find that the global-batch LBL strategy yields excellent performance gains in
both pre-training perplexity and downstream tasks. Our analysis reveals that
the global-batch LBL also greatly improves the domain specialization of MoE
experts.
679071da11a3f67d8f498649
Zihan Qiu
admin_assigned
2025-01-22T15:32:39.315Z
647ccbd6e07cf9bb2d485244
/avatars/e8915abaff04f6762247e196b7cf84df.svg
Zihan Qiu
user
QwQZh
679071da11a3f67d8f49864a
Zeyu Huang
claimed_verified
2025-01-22T15:35:10.842Z
64a5520dea3c5861303064b0
/avatars/7e618bcfc4d8d3291cf9bc8f2c4c1b15.svg
Zeyu hUANG
user
zeroyhuang
679071da11a3f67d8f49864b
Bo Zheng
679071da11a3f67d8f49864c
Kaiyue Wen
claimed_verified
2025-01-22T20:49:51.765Z
6690a44e76c0fa097f9f65c9
/avatars/0c6efa10889d1b6dd9a3f9985f4d2d97.svg
Kaiyue Wen
user
KaiyueWen
679071da11a3f67d8f49864d
Zekun Wang
admin_assigned
2025-01-22T15:31:36.863Z
656832dfbd65fd41ee7aa8cd
https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
Zekun Wang
user
kugwzk
679071da11a3f67d8f49864e
Rui Men
679071da11a3f67d8f49864f
Ivan Titov
admin_assigned
2025-01-22T15:30:57.878Z
666f6375623133f1ce79c021
/avatars/b57a457cff81d685fdd9eadee53445fc.svg
Ivan Titov
user
Ivanchoo
679071da11a3f67d8f498650
Dayiheng Liu
claimed_verified
2025-01-22T10:08:15.815Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
679071da11a3f67d8f498651
Jingren Zhou
admin_assigned
2025-01-22T15:30:48.272Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
679071da11a3f67d8f498652
Junyang Lin
admin_assigned
2025-01-22T15:30:37.427Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
Demons in the Detail: On Implementing Load Balancing Loss for Training
  Specialized Mixture-of-Expert Models
This paper revisits the implementation of
Load-balancing Loss (LBL) when training
Mixture-of-Experts (MoEs) models. Specifically, LBL for MoEs is defined as N_E
sum_{i=1}^{N_E} f_i p_i, where N_E is the total number of experts, f_i
represents the frequency of expert i being selected, and p_i denotes the
average gating score of the expert i. Existing MoE training frameworks
usually employ the parallel training strategy so that f_i and the LBL are
calculated within a micro-batch and then averaged across parallel
groups. In essence, a micro-batch for training billion-scale LLMs normally
contains very few sequences. So, the micro-batch LBL is almost at the sequence
level, and the router is pushed to distribute the token evenly within each
sequence. Under this strict constraint, even tokens from a domain-specific
sequence (e.g., code) are uniformly routed to all experts, thereby
inhibiting expert specialization. In this work, we propose calculating LBL
using a global-batch to loose this constraint. Because a
global-batch contains much more diverse sequences than a micro-batch, which
will encourage load balance at the corpus level. Specifically, we introduce an
extra communication step to synchronize f_i across micro-batches and then use
it to calculate the LBL. Through experiments on training MoEs-based LLMs (up to
42.8B total parameters and 400B tokens), we surprisingly
find that the global-batch LBL strategy yields excellent performance gains in
both pre-training perplexity and downstream tasks. Our analysis reveals that
the global-batch LBL also greatly improves the domain specialization of MoE
experts.
Calculating Load-Balancing Loss using a global-batch instead of a micro-batch improves performance and domain specialization in MoE-based large language models.
Load-Balancing Loss (LBL) Mixture-of-Experts (MoEs) micro-batch global-batch expert specialization pre-training perplexity downstream tasks