title: Mixture-of-Transformers: A Sparse and Scalable Architecture for
    Multi-Modal Foundation Models
summary: The development of large language models (LLMs) has expanded to multi-modal
  systems capable of processing text, images, and speech within a unified
  framework. Training these models demands significantly larger datasets and
  computational resources compared to text-only LLMs. To address the scaling
  challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal
  transformer architecture that significantly reduces pretraining computational
  costs. MoT decouples non-embedding parameters of the model by modality --
  including feed-forward networks, attention matrices, and layer normalization --
  enabling modality-specific processing with global self-attention over the full
  input sequence. We evaluate MoT across multiple settings and model scales. In
  the Chameleon 7B setting (autoregressive text-and-image generation), MoT
  matches the dense baseline's performance using only 55.8\% of the FLOPs. When
  extended to include speech, MoT reaches speech performance comparable to the
  dense baseline with only 37.2\% of the FLOPs. In the Transfusion setting, where
  text and image are trained with different objectives, a 7B MoT model matches
  the image modality performance of the dense baseline with one third of the
  FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image
  generation metrics. System profiling further highlights MoT's practical
  benefits, achieving dense baseline image quality in 47.2\% of the wall-clock
  time and text quality in 75.6\% of the wall-clock time (measured on AWS
  p4de.24xlarge instances with NVIDIA A100 GPUs).
paper_authors._id: 672d87680c2b474d28e60295
paper_authors.name: Weixin Liang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-08T03:37:15.688Z
paper_authors._id: 630da4333dc31beba6f626b2
paper_authors.avatarUrl: /avatars/b3b3cf804e58a820981c27c5272d4570.svg
paper_authors.fullname: Wx
paper_authors.type: user
paper_authors.user: WxWx
paper_authors._id: 672d87680c2b474d28e60296
paper_authors.name: Lili Yu
paper_authors._id: 672d87680c2b474d28e60297
paper_authors.name: Liang Luo
paper_authors._id: 672d87680c2b474d28e60298
paper_authors.name: Srinivasan Iyer
paper_authors._id: 672d87680c2b474d28e60299
paper_authors.name: Ning Dong
paper_authors._id: 672d87680c2b474d28e6029a
paper_authors.name: Chunting Zhou
paper_authors._id: 672d87680c2b474d28e6029b
paper_authors.name: Gargi Ghosh
paper_authors._id: 672d87680c2b474d28e6029c
paper_authors.name: Mike Lewis
paper_authors._id: 672d87680c2b474d28e6029d
paper_authors.name: Wen-tau Yih
paper_authors._id: 672d87680c2b474d28e6029e
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 672d87680c2b474d28e6029f
paper_authors.name: Xi Victoria Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:51:16.868Z
paper_authors._id: 61df349e34b589fbaf2e93c4
paper_authors.avatarUrl: /avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
paper_authors.fullname: Victoria Lin
paper_authors.type: user
paper_authors.user: VictoriaLinML
paper_title: Mixture-of-Transformers: A Sparse and Scalable Architecture for
    Multi-Modal Foundation Models
paper_summary: The development of large language models (LLMs) has expanded to multi-modal
  systems capable of processing text, images, and speech within a unified
  framework. Training these models demands significantly larger datasets and
  computational resources compared to text-only LLMs. To address the scaling
  challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal
  transformer architecture that significantly reduces pretraining computational
  costs. MoT decouples non-embedding parameters of the model by modality --
  including feed-forward networks, attention matrices, and layer normalization --
  enabling modality-specific processing with global self-attention over the full
  input sequence. We evaluate MoT across multiple settings and model scales. In
  the Chameleon 7B setting (autoregressive text-and-image generation), MoT
  matches the dense baseline's performance using only 55.8\% of the FLOPs. When
  extended to include speech, MoT reaches speech performance comparable to the
  dense baseline with only 37.2\% of the FLOPs. In the Transfusion setting, where
  text and image are trained with different objectives, a 7B MoT model matches
  the image modality performance of the dense baseline with one third of the
  FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image
  generation metrics. System profiling further highlights MoT's practical
  benefits, achieving dense baseline image quality in 47.2\% of the wall-clock
  time and text quality in 75.6\% of the wall-clock time (measured on AWS
  p4de.24xlarge instances with NVIDIA A100 GPUs).
paper_ai_summary: Mixture-of-Transformers (MoT) reduces pretraining computational costs for multi-modal large language models by decoupling non-embedding parameters by modality, enabling efficient processing and competitive performance across text, image, and speech.
paper_ai_keywords: Mixture-of-Transformers MoT multi-modal transformer feed-forward networks attention matrices layer normalization global self-attention Chameleon autoregressive text-and-image generation speech Transfusion image generation metrics FLOPs wall-clock time AWS p4de.24xlarge NVIDIA A100 GPUs