From Sparse to Soft Mixtures of Experts
Sparse mixture of expert architectures (MoEs) scale model capacity without
large increases in training or inference costs. Despite their success, MoEs
suffer from a number of issues: training instability, token dropping, inability
to scale the number of experts, or ineffective finetuning. In this work, we
proposeSoft MoE, a fully-differentiable sparse Transformer that addresses these
challenges, while maintaining the benefits of MoEs. Soft MoE performs an
implicit soft assignment by passing different weighted combinations of all
input tokens to each expert. As in other MoE works, experts in Soft MoE only
process a subset of the (combined) tokens, enabling larger model capacity at
lower inference cost. In the context of visual recognition, Soft MoE greatly
outperforms standard Transformers (ViTs) and popular MoE variants (Tokens
Choice and Experts Choice). For example, Soft MoE-Base/16 requires 10.5x lower
inference cost (5.7x lower wall-clock time) than ViT-Huge/14 while matching its
performance after similar training. Soft MoE also scales well: Soft MoE Huge/14
with 128 experts in 16 MoE layers has over 40x more parameters than ViT
Huge/14, while inference time cost grows by only 2%, and it performs
substantially better.
64cb0c431f8a37e3ed338fe8
Joan Puigcerver
claimed_verified
2023-08-03T09:47:17.207Z
64afea8efd620c8a7ad4ebd7
https://cdn-avatars.huggingface.co/v1/production/uploads/64afea8efd620c8a7ad4ebd7/O3e9nLirWFVZ_jMN4lFdH.jpeg
Joan Puigcerver
user
joapuipe
64cb0c431f8a37e3ed338fe9
Carlos Riquelme
64cb0c431f8a37e3ed338fea
Basil Mustafa
64cb0c431f8a37e3ed338feb
Neil Houlsby
From Sparse to Soft Mixtures of Experts
Sparse mixture of expert architectures (MoEs) scale model capacity without
large increases in training or inference costs. Despite their success, MoEs
suffer from a number of issues: training instability, token dropping, inability
to scale the number of experts, or ineffective finetuning. In this work, we
proposeSoft MoE, a fully-differentiable sparse Transformer that addresses these
challenges, while maintaining the benefits of MoEs. Soft MoE performs an
implicit soft assignment by passing different weighted combinations of all
input tokens to each expert. As in other MoE works, experts in Soft MoE only
process a subset of the (combined) tokens, enabling larger model capacity at
lower inference cost. In the context of visual recognition, Soft MoE greatly
outperforms standard Transformers (ViTs) and popular MoE variants (Tokens
Choice and Experts Choice). For example, Soft MoE-Base/16 requires 10.5x lower
inference cost (5.7x lower wall-clock time) than ViT-Huge/14 while matching its
performance after similar training. Soft MoE also scales well: Soft MoE Huge/14
with 128 experts in 16 MoE layers has over 40x more parameters than ViT
Huge/14, while inference time cost grows by only 2%, and it performs
substantially better.
Soft MoE, a differentiable sparse Transformer, stabilizes training, reduces inference cost, and outperforms traditional Transformers and MoE variants in visual recognition.
sparse mixture of expert architectures MoEs training instability token dropping Soft MoE implicit soft assignment fully-differentiable sparse Transformer visual recognition Soft MoE-Base Soft MoE Huge ViT-Huge parameter efficiency inference cost performance