title: Alternating Gradient Descent and Mixture-of-Experts for Integrated
    Multimodal Perception
summary: We present Integrated Multimodal Perception (IMP), a simple and scalable
  multimodal multi-task training and modeling approach. IMP integrates multimodal
  inputs including image, video, text, and audio into a single Transformer
  encoder with minimal modality-specific components. IMP makes use of a novel
  design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts
  (MoE) for efficient model \& task scaling. We conduct extensive empirical
  studies about IMP and reveal the following key insights: 1) performing gradient
  descent updates by alternating on diverse heterogeneous modalities, loss
  functions, and tasks, while also varying input resolutions, efficiently
  improves multimodal understanding. 2) model sparsification with MoE on a single
  modality-agnostic encoder substantially improves the performance, outperforming
  dense models that use modality-specific encoders or additional fusion layers
  and greatly mitigating the conflicts between modalities. IMP achieves
  competitive performance on a wide range of downstream tasks including image
  classification, video classification, image-text, and video-text retrieval.
  Most notably, we train a sparse IMP-MoE-L focusing on video tasks that achieves
  new state-of-the-art in zero-shot video classification. Our model achieves
  77.0% on Kinetics-400, 76.8% on Kinetics-600, and 76.8% on Kinetics-700
  zero-shot classification accuracy, improving the previous state-of-the-art by
  +5%, +6.7%, and +5.8%, respectively, while using only 15% of their total
  training computational cost.
paper_authors._id: 645c5dfe23ed9b7788d6e478
paper_authors.name: Hassan Akbari
paper_authors._id: 645c5dfe23ed9b7788d6e479
paper_authors.name: Dan Kondratyuk
paper_authors._id: 645c5dfe23ed9b7788d6e47a
paper_authors.name: Yin Cui
paper_authors._id: 645c5dfe23ed9b7788d6e47b
paper_authors.name: Rachel Hornung
paper_authors._id: 645c5dfe23ed9b7788d6e47c
paper_authors.name: Huisheng Wang
paper_authors._id: 645c5dfe23ed9b7788d6e47d
paper_authors.name: Hartwig Adam
paper_title: Alternating Gradient Descent and Mixture-of-Experts for Integrated
    Multimodal Perception
paper_summary: We present Integrated Multimodal Perception (IMP), a simple and scalable
  multimodal multi-task training and modeling approach. IMP integrates multimodal
  inputs including image, video, text, and audio into a single Transformer
  encoder with minimal modality-specific components. IMP makes use of a novel
  design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts
  (MoE) for efficient model \& task scaling. We conduct extensive empirical
  studies about IMP and reveal the following key insights: 1) performing gradient
  descent updates by alternating on diverse heterogeneous modalities, loss
  functions, and tasks, while also varying input resolutions, efficiently
  improves multimodal understanding. 2) model sparsification with MoE on a single
  modality-agnostic encoder substantially improves the performance, outperforming
  dense models that use modality-specific encoders or additional fusion layers
  and greatly mitigating the conflicts between modalities. IMP achieves
  competitive performance on a wide range of downstream tasks including image
  classification, video classification, image-text, and video-text retrieval.
  Most notably, we train a sparse IMP-MoE-L focusing on video tasks that achieves
  new state-of-the-art in zero-shot video classification. Our model achieves
  77.0% on Kinetics-400, 76.8% on Kinetics-600, and 76.8% on Kinetics-700
  zero-shot classification accuracy, improving the previous state-of-the-art by
  +5%, +6.7%, and +5.8%, respectively, while using only 15% of their total
  training computational cost.
paper_ai_keywords: Alternating Gradient Descent (AGD) Mixture-of-experts (MoE) IMP-MoE-L multimodal understanding model sparsification image classification video classification image-text retrieval video-text retrieval zero-shot video classification Kinetics-400 Kinetics-600 Kinetics-700