Alternating Gradient Descent and Mixture-of-Experts for Integrated
  Multimodal Perception
We present Integrated Multimodal Perception (IMP), a simple and scalable
multimodal multi-task training and modeling approach. IMP integrates multimodal
inputs including image, video, text, and audio into a single Transformer
encoder with minimal modality-specific components. IMP makes use of a novel
design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts
(MoE) for efficient model \& task scaling. We conduct extensive empirical
studies about IMP and reveal the following key insights: 1) performing gradient
descent updates by alternating on diverse heterogeneous modalities, loss
functions, and tasks, while also varying input resolutions, efficiently
improves multimodal understanding. 2) model sparsification with MoE on a single
modality-agnostic encoder substantially improves the performance, outperforming
dense models that use modality-specific encoders or additional fusion layers
and greatly mitigating the conflicts between modalities. IMP achieves
competitive performance on a wide range of downstream tasks including image
classification, video classification, image-text, and video-text retrieval.
Most notably, we train a sparse IMP-MoE-L focusing on video tasks that achieves
new state-of-the-art in zero-shot video classification. Our model achieves
77.0% on Kinetics-400, 76.8% on Kinetics-600, and 76.8% on Kinetics-700
zero-shot classification accuracy, improving the previous state-of-the-art by
+5%, +6.7%, and +5.8%, respectively, while using only 15% of their total
training computational cost.
645c5dfe23ed9b7788d6e478
Hassan Akbari
645c5dfe23ed9b7788d6e479
Dan Kondratyuk
645c5dfe23ed9b7788d6e47a
Yin Cui
claimed_verified
2025-03-24T08:57:19.809Z
649f05367b57fab3a5b27c8b
https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
Yin Cui
user
richardaecn
645c5dfe23ed9b7788d6e47b
Rachel Hornung
645c5dfe23ed9b7788d6e47c
Huisheng Wang
645c5dfe23ed9b7788d6e47d
Hartwig Adam
Alternating Gradient Descent and Mixture-of-Experts for Integrated
  Multimodal Perception
We present Integrated Multimodal Perception (IMP), a simple and scalable
multimodal multi-task training and modeling approach. IMP integrates multimodal
inputs including image, video, text, and audio into a single Transformer
encoder with minimal modality-specific components. IMP makes use of a novel
design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts
(MoE) for efficient model \& task scaling. We conduct extensive empirical
studies about IMP and reveal the following key insights: 1) performing gradient
descent updates by alternating on diverse heterogeneous modalities, loss
functions, and tasks, while also varying input resolutions, efficiently
improves multimodal understanding. 2) model sparsification with MoE on a single
modality-agnostic encoder substantially improves the performance, outperforming
dense models that use modality-specific encoders or additional fusion layers
and greatly mitigating the conflicts between modalities. IMP achieves
competitive performance on a wide range of downstream tasks including image
classification, video classification, image-text, and video-text retrieval.
Most notably, we train a sparse IMP-MoE-L focusing on video tasks that achieves
new state-of-the-art in zero-shot video classification. Our model achieves
77.0% on Kinetics-400, 76.8% on Kinetics-600, and 76.8% on Kinetics-700
zero-shot classification accuracy, improving the previous state-of-the-art by
+5%, +6.7%, and +5.8%, respectively, while using only 15% of their total
training computational cost.
Integrated Multimodal Perception (IMP) combines Alternating Gradient Descent and Mixture-of-Experts in a single Transformer encoder to improve multimodal understanding and achieve state-of-the-art zero-shot video classification.
Integrated Multimodal Perception IMP Transformer encoder Alternating Gradient Descent AGD Mixture-of-Experts MoE multimodal understanding image classification video classification image-text retrieval video-text retrieval zero-shot video classification Kinetics-400 Kinetics-600 Kinetics-700