title: Mixture of Nested Experts: Adaptive Processing of Visual Tokens
summary: The visual medium (images and videos) naturally contains a large amount of
  information redundancy, thereby providing a great opportunity for leveraging
  efficiency in processing. While Vision Transformer (ViT) based models scale
  effectively to large data regimes, they fail to capitalize on this inherent
  redundancy, leading to higher computational costs. Mixture of Experts (MoE)
  networks demonstrate scalability while maintaining same inference-time costs,
  but they come with a larger parameter footprint. We present Mixture of Nested
  Experts (MoNE), which utilizes a nested structure for experts, wherein
  individual experts fall on an increasing compute-accuracy curve. Given a
  compute budget, MoNE learns to dynamically choose tokens in a priority order,
  and thus redundant tokens are processed through cheaper nested experts. Using
  this framework, we achieve equivalent performance as the baseline models, while
  reducing inference time compute by over two-fold. We validate our approach on
  standard image and video datasets - ImageNet-21K, Kinetics400, and
  Something-Something-v2. We further highlight MoNE's adaptability by
  showcasing its ability to maintain strong performance across different
  inference-time compute budgets on videos, using only a single trained model.
paper_authors._id: 66a8603d2f11aaf660edf44b
paper_authors.name: Gagan Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T08:16:10.660Z
paper_authors._id: 6275d60b7128d87e9f9452c0
paper_authors.avatarUrl: /avatars/dd04bd7c3684219410faab6351acafc7.svg
paper_authors.fullname: Gagan Jain
paper_authors.type: user
paper_authors.user: smileysky
paper_authors._id: 66a8603d2f11aaf660edf44c
paper_authors.name: Nidhi Hegde
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:40:01.166Z
paper_authors._id: 666a6ba3ae0b4ceca4f1026e
paper_authors.avatarUrl: /avatars/e7a2e2369f11aef773dae16ab7aee558.svg
paper_authors.fullname: nidhi hegde
paper_authors.type: user
paper_authors.user: nidnap
paper_authors._id: 66a8603d2f11aaf660edf44d
paper_authors.name: Aditya Kusupati
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:40:06.738Z
paper_authors._id: 65dd3b03f0f26b277be1e91e
paper_authors.avatarUrl: /avatars/67160c2d5241aea5c5d9b211f21f9caa.svg
paper_authors.fullname: Aditya Kusupati
paper_authors.type: user
paper_authors.user: adityakusupati
paper_authors._id: 66a8603d2f11aaf660edf44e
paper_authors.name: Arsha Nagrani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:40:11.631Z
paper_authors._id: 65202b027bf8cc2dd27d8c4b
paper_authors.avatarUrl: /avatars/22504800e86e112f3387eae08026e8fa.svg
paper_authors.fullname: Arsha Nagrani 
paper_authors.type: user
paper_authors.user: arsha1
paper_authors._id: 66a8603d2f11aaf660edf44f
paper_authors.name: Shyamal Buch
paper_authors._id: 66a8603d2f11aaf660edf450
paper_authors.name: Prateek Jain
paper_authors._id: 66a8603d2f11aaf660edf451
paper_authors.name: Anurag Arnab
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:40:45.926Z
paper_authors._id: 63d7f0466ac3104e50c5311e
paper_authors.avatarUrl: /avatars/2421bfb7747664d1acefbac1c00c3475.svg
paper_authors.fullname: Anurag Arnab
paper_authors.type: user
paper_authors.user: anurag-arnab
paper_authors._id: 66a8603d2f11aaf660edf452
paper_authors.name: Sujoy Paul
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T08:44:10.376Z
paper_authors._id: 66a8a7968268a5b1a0819668
paper_authors.avatarUrl: /avatars/a8cdca874279beeef1023c8ecb10403a.svg
paper_authors.fullname: Sujoy Paul
paper_authors.type: user
paper_authors.user: sujoyp
paper_title: Mixture of Nested Experts: Adaptive Processing of Visual Tokens
paper_summary: The visual medium (images and videos) naturally contains a large amount of
  information redundancy, thereby providing a great opportunity for leveraging
  efficiency in processing. While Vision Transformer (ViT) based models scale
  effectively to large data regimes, they fail to capitalize on this inherent
  redundancy, leading to higher computational costs. Mixture of Experts (MoE)
  networks demonstrate scalability while maintaining same inference-time costs,
  but they come with a larger parameter footprint. We present Mixture of Nested
  Experts (MoNE), which utilizes a nested structure for experts, wherein
  individual experts fall on an increasing compute-accuracy curve. Given a
  compute budget, MoNE learns to dynamically choose tokens in a priority order,
  and thus redundant tokens are processed through cheaper nested experts. Using
  this framework, we achieve equivalent performance as the baseline models, while
  reducing inference time compute by over two-fold. We validate our approach on
  standard image and video datasets - ImageNet-21K, Kinetics400, and
  Something-Something-v2. We further highlight MoNE's adaptability by
  showcasing its ability to maintain strong performance across different
  inference-time compute budgets on videos, using only a single trained model.
paper_ai_keywords: Vision Transformer (ViT) Mixture of Experts (MoE) Mixture of Nested Experts (MoNE) compute-accuracy curve dynamic token prioritization nested structure inference-time compute ImageNet-21K Kinetics400 Something-Something-v2 adaptability