title: From Sparse to Soft Mixtures of Experts
summary: Sparse mixture of expert architectures (MoEs) scale model capacity without
  large increases in training or inference costs. Despite their success, MoEs
  suffer from a number of issues: training instability, token dropping, inability
  to scale the number of experts, or ineffective finetuning. In this work, we
  proposeSoft MoE, a fully-differentiable sparse Transformer that addresses these
  challenges, while maintaining the benefits of MoEs. Soft MoE performs an
  implicit soft assignment by passing different weighted combinations of all
  input tokens to each expert. As in other MoE works, experts in Soft MoE only
  process a subset of the (combined) tokens, enabling larger model capacity at
  lower inference cost. In the context of visual recognition, Soft MoE greatly
  outperforms standard Transformers (ViTs) and popular MoE variants (Tokens
  Choice and Experts Choice). For example, Soft MoE-Base/16 requires 10.5x lower
  inference cost (5.7x lower wall-clock time) than ViT-Huge/14 while matching its
  performance after similar training. Soft MoE also scales well: Soft MoE Huge/14
  with 128 experts in 16 MoE layers has over 40x more parameters than ViT
  Huge/14, while inference time cost grows by only 2%, and it performs
  substantially better.
paper_authors._id: 64cb0c431f8a37e3ed338fe8
paper_authors.name: Joan Puigcerver
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-03T09:47:17.207Z
paper_authors._id: 64afea8efd620c8a7ad4ebd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64afea8efd620c8a7ad4ebd7/O3e9nLirWFVZ_jMN4lFdH.jpeg
paper_authors.fullname: Joan Puigcerver
paper_authors.type: user
paper_authors.user: joapuipe
paper_authors._id: 64cb0c431f8a37e3ed338fe9
paper_authors.name: Carlos Riquelme
paper_authors._id: 64cb0c431f8a37e3ed338fea
paper_authors.name: Basil Mustafa
paper_authors._id: 64cb0c431f8a37e3ed338feb
paper_authors.name: Neil Houlsby
paper_title: From Sparse to Soft Mixtures of Experts
paper_summary: Sparse mixture of expert architectures (MoEs) scale model capacity without
  large increases in training or inference costs. Despite their success, MoEs
  suffer from a number of issues: training instability, token dropping, inability
  to scale the number of experts, or ineffective finetuning. In this work, we
  proposeSoft MoE, a fully-differentiable sparse Transformer that addresses these
  challenges, while maintaining the benefits of MoEs. Soft MoE performs an
  implicit soft assignment by passing different weighted combinations of all
  input tokens to each expert. As in other MoE works, experts in Soft MoE only
  process a subset of the (combined) tokens, enabling larger model capacity at
  lower inference cost. In the context of visual recognition, Soft MoE greatly
  outperforms standard Transformers (ViTs) and popular MoE variants (Tokens
  Choice and Experts Choice). For example, Soft MoE-Base/16 requires 10.5x lower
  inference cost (5.7x lower wall-clock time) than ViT-Huge/14 while matching its
  performance after similar training. Soft MoE also scales well: Soft MoE Huge/14
  with 128 experts in 16 MoE layers has over 40x more parameters than ViT
  Huge/14, while inference time cost grows by only 2%, and it performs
  substantially better.
paper_ai_summary: Soft MoE, a differentiable sparse Transformer, stabilizes training, reduces inference cost, and outperforms traditional Transformers and MoE variants in visual recognition.
paper_ai_keywords: sparse mixture of expert architectures MoEs training instability token dropping Soft MoE implicit soft assignment fully-differentiable sparse Transformer visual recognition Soft MoE-Base Soft MoE Huge ViT-Huge parameter efficiency inference cost performance