title: Mobile V-MoEs: Scaling Down Vision Transformers via Sparse
    Mixture-of-Experts
summary: Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
  to their ability to decouple model size from inference efficiency by only
  activating a small subset of the model parameters for any given input token. As
  such, sparse MoEs have enabled unprecedented scalability, resulting in
  tremendous successes across domains such as natural language processing and
  computer vision. In this work, we instead explore the use of sparse MoEs to
  scale-down Vision Transformers (ViTs) to make them more attractive for
  resource-constrained vision applications. To this end, we propose a simplified
  and mobile-friendly MoE design where entire images rather than individual
  patches are routed to the experts. We also propose a stable MoE training
  procedure that uses super-class information to guide the router. We empirically
  show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off
  between performance and efficiency than the corresponding dense ViTs. For
  example, for the ViT-Tiny model, our Mobile V-MoE outperforms its dense
  counterpart by 3.39% on ImageNet-1k. For an even smaller ViT variant with only
  54M FLOPs inference cost, our MoE achieves an improvement of 4.66%.
paper_authors._id: 64fe72b048411fc789d43764
paper_authors.name: Erik Daxberger
paper_authors._id: 64fe72b048411fc789d43765
paper_authors.name: Floris Weers
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T11:02:53.284Z
paper_authors._id: 65f41f1d6724ea19d7bc9052
paper_authors.avatarUrl: /avatars/4ccd6279aac3a18e1b23642a1a549e2f.svg
paper_authors.fullname: Floris Weers
paper_authors.type: user
paper_authors.user: fweers
paper_authors._id: 64fe72b048411fc789d43766
paper_authors.name: Bowen Zhang
paper_authors._id: 64fe72b048411fc789d43767
paper_authors.name: Tom Gunter
paper_authors._id: 64fe72b048411fc789d43768
paper_authors.name: Ruoming Pang
paper_authors._id: 64fe72b048411fc789d43769
paper_authors.name: Marcin Eichner
paper_authors._id: 64fe72b048411fc789d4376a
paper_authors.name: Michael Emmersberger
paper_authors._id: 64fe72b048411fc789d4376b
paper_authors.name: Yinfei Yang
paper_authors._id: 64fe72b048411fc789d4376c
paper_authors.name: Alexander Toshev
paper_authors._id: 64fe72b048411fc789d4376d
paper_authors.name: Xianzhi Du
paper_title: Mobile V-MoEs: Scaling Down Vision Transformers via Sparse
    Mixture-of-Experts
paper_summary: Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
  to their ability to decouple model size from inference efficiency by only
  activating a small subset of the model parameters for any given input token. As
  such, sparse MoEs have enabled unprecedented scalability, resulting in
  tremendous successes across domains such as natural language processing and
  computer vision. In this work, we instead explore the use of sparse MoEs to
  scale-down Vision Transformers (ViTs) to make them more attractive for
  resource-constrained vision applications. To this end, we propose a simplified
  and mobile-friendly MoE design where entire images rather than individual
  patches are routed to the experts. We also propose a stable MoE training
  procedure that uses super-class information to guide the router. We empirically
  show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off
  between performance and efficiency than the corresponding dense ViTs. For
  example, for the ViT-Tiny model, our Mobile V-MoE outperforms its dense
  counterpart by 3.39% on ImageNet-1k. For an even smaller ViT variant with only
  54M FLOPs inference cost, our MoE achieves an improvement of 4.66%.
paper_ai_summary: Sparse Mobile Vision MoEs are applied to reduce resource requirements in Vision Transformers, achieving improved performance and efficiency compared to dense models.
paper_ai_keywords: Sparse Mixture-of-Experts Visual Transformers MoE ViTs image routing super-class information router Mobile Vision MoEs V-MoEs