Mixture of Nested Experts: Adaptive Processing of Visual Tokens
The visual medium (images and videos) naturally contains a large amount of
information redundancy, thereby providing a great opportunity for leveraging
efficiency in processing. While Vision Transformer (ViT) based models scale
effectively to large data regimes, they fail to capitalize on this inherent
redundancy, leading to higher computational costs. Mixture of Experts (MoE)
networks demonstrate scalability while maintaining same inference-time costs,
but they come with a larger parameter footprint. We present Mixture of Nested
Experts (MoNE), which utilizes a nested structure for experts, wherein
individual experts fall on an increasing compute-accuracy curve. Given a
compute budget, MoNE learns to dynamically choose tokens in a priority order,
and thus redundant tokens are processed through cheaper nested experts. Using
this framework, we achieve equivalent performance as the baseline models, while
reducing inference time compute by over two-fold. We validate our approach on
standard image and video datasets - ImageNet-21K, Kinetics400, and
Something-Something-v2. We further highlight MoNE's adaptability by
showcasing its ability to maintain strong performance across different
inference-time compute budgets on videos, using only a single trained model.
66a8603d2f11aaf660edf44b
Gagan Jain
claimed_verified
2024-07-30T08:16:10.660Z
6275d60b7128d87e9f9452c0
/avatars/dd04bd7c3684219410faab6351acafc7.svg
Gagan Jain
user
smileysky
66a8603d2f11aaf660edf44c
Nidhi Hegde
admin_assigned
2024-07-30T08:40:01.166Z
666a6ba3ae0b4ceca4f1026e
/avatars/e7a2e2369f11aef773dae16ab7aee558.svg
nidhi hegde
user
nidnap
66a8603d2f11aaf660edf44d
Aditya Kusupati
admin_assigned
2024-07-30T08:40:06.738Z
65dd3b03f0f26b277be1e91e
/avatars/67160c2d5241aea5c5d9b211f21f9caa.svg
Aditya Kusupati
user
adityakusupati
66a8603d2f11aaf660edf44e
Arsha Nagrani
admin_assigned
2024-07-30T08:40:11.631Z
65202b027bf8cc2dd27d8c4b
/avatars/22504800e86e112f3387eae08026e8fa.svg
Arsha Nagrani 
user
arsha1
66a8603d2f11aaf660edf44f
Shyamal Buch
66a8603d2f11aaf660edf450
Prateek Jain
66a8603d2f11aaf660edf451
Anurag Arnab
admin_assigned
2024-07-30T08:40:45.926Z
63d7f0466ac3104e50c5311e
/avatars/2421bfb7747664d1acefbac1c00c3475.svg
Anurag Arnab
user
anurag-arnab
66a8603d2f11aaf660edf452
Sujoy Paul
claimed_verified
2024-07-30T08:44:10.376Z
66a8a7968268a5b1a0819668
/avatars/a8cdca874279beeef1023c8ecb10403a.svg
Sujoy Paul
user
sujoyp
Mixture of Nested Experts: Adaptive Processing of Visual Tokens
The visual medium (images and videos) naturally contains a large amount of
information redundancy, thereby providing a great opportunity for leveraging
efficiency in processing. While Vision Transformer (ViT) based models scale
effectively to large data regimes, they fail to capitalize on this inherent
redundancy, leading to higher computational costs. Mixture of Experts (MoE)
networks demonstrate scalability while maintaining same inference-time costs,
but they come with a larger parameter footprint. We present Mixture of Nested
Experts (MoNE), which utilizes a nested structure for experts, wherein
individual experts fall on an increasing compute-accuracy curve. Given a
compute budget, MoNE learns to dynamically choose tokens in a priority order,
and thus redundant tokens are processed through cheaper nested experts. Using
this framework, we achieve equivalent performance as the baseline models, while
reducing inference time compute by over two-fold. We validate our approach on
standard image and video datasets - ImageNet-21K, Kinetics400, and
Something-Something-v2. We further highlight MoNE's adaptability by
showcasing its ability to maintain strong performance across different
inference-time compute budgets on videos, using only a single trained model.
Mixture of Nested Experts (MoNE) reduces inference time by dynamically processing tokens through nested experts, achieving performance equivalent to baseline models.
Vision Transformer Mixture of Experts MoNE compute-accuracy curve inference-time costs parameter footprint nested structure dynamic token selection ImageNet-21K Kinetics400 Something-Something-v2