title: MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware
    Experts
summary: We introduce MoMa, a novel modality-aware mixture-of-experts (MoE)
  architecture designed for pre-training mixed-modal, early-fusion language
  models. MoMa processes images and text in arbitrary sequences by dividing
  expert modules into modality-specific groups. These groups exclusively process
  designated tokens while employing learned routing within each group to maintain
  semantically informed adaptivity. Our empirical results reveal substantial
  pre-training efficiency gains through this modality-specific parameter
  allocation. Under a 1-trillion-token training budget, the MoMa 1.4B model,
  featuring 4 text experts and 4 image experts, achieves impressive FLOPs
  savings: 3.7x overall, with 2.6x for text and 5.2x for image processing
  compared to a compute-equivalent dense baseline, measured by pre-training loss.
  This outperforms the standard expert-choice MoE with 8 mixed-modal experts,
  which achieves 3x overall FLOPs savings (3x for text, 2.8x for image).
  Combining MoMa with mixture-of-depths (MoD) further improves pre-training FLOPs
  savings to 4.2x overall (text: 3.4x, image: 5.3x), although this combination
  hurts performance in causal inference due to increased sensitivity to router
  accuracy. These results demonstrate MoMa's potential to significantly advance
  the efficiency of mixed-modal, early-fusion language model pre-training, paving
  the way for more resource-efficient and capable multimodal AI systems.
paper_authors._id: 66aaeffdadfd339ee6e4a0d8
paper_authors.name: Xi Victoria Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T15:06:16.361Z
paper_authors._id: 61df349e34b589fbaf2e93c4
paper_authors.avatarUrl: /avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
paper_authors.fullname: Victoria Lin
paper_authors.type: user
paper_authors.user: VictoriaLinML
paper_authors._id: 66aaeffdadfd339ee6e4a0d9
paper_authors.name: Akshat Shrivastava
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-01T02:16:30.719Z
paper_authors._id: 64b6d105f616b195a39f462a
paper_authors.avatarUrl: /avatars/f33805c3e42e476a135c3cf869cc423a.svg
paper_authors.fullname: Akshat
paper_authors.type: user
paper_authors.user: akshats07
paper_authors._id: 66aaeffdadfd339ee6e4a0da
paper_authors.name: Liang Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T15:06:36.409Z
paper_authors._id: 65e8be4b45ae4e903cee573a
paper_authors.avatarUrl: /avatars/546e503dc13a1c081c6b139f551b5b00.svg
paper_authors.fullname: Liang Luo
paper_authors.type: user
paper_authors.user: liangluo
paper_authors._id: 66aaeffdadfd339ee6e4a0db
paper_authors.name: Srinivasan Iyer
paper_authors._id: 66aaeffdadfd339ee6e4a0dc
paper_authors.name: Mike Lewis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T15:07:10.452Z
paper_authors._id: 5e67eda71009063689407484
paper_authors.avatarUrl: /avatars/6d7688faba888d463584eaa02ad011e3.svg
paper_authors.fullname: Mike Lewis
paper_authors.type: user
paper_authors.user: mikelewis0
paper_authors._id: 66aaeffdadfd339ee6e4a0dd
paper_authors.name: Gargi Gosh
paper_authors._id: 66aaeffdadfd339ee6e4a0de
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 66aaeffdadfd339ee6e4a0df
paper_authors.name: Armen Aghajanyan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-24T21:25:49.162Z
paper_authors._id: 671abaf6e0813cc6983103c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/SLytMjnVS452F9qhF_GFQ.png
paper_authors.fullname: Armen Aghajanyan
paper_authors.type: user
paper_authors.user: ArmenAgh
paper_title: MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware
    Experts
paper_summary: We introduce MoMa, a novel modality-aware mixture-of-experts (MoE)
  architecture designed for pre-training mixed-modal, early-fusion language
  models. MoMa processes images and text in arbitrary sequences by dividing
  expert modules into modality-specific groups. These groups exclusively process
  designated tokens while employing learned routing within each group to maintain
  semantically informed adaptivity. Our empirical results reveal substantial
  pre-training efficiency gains through this modality-specific parameter
  allocation. Under a 1-trillion-token training budget, the MoMa 1.4B model,
  featuring 4 text experts and 4 image experts, achieves impressive FLOPs
  savings: 3.7x overall, with 2.6x for text and 5.2x for image processing
  compared to a compute-equivalent dense baseline, measured by pre-training loss.
  This outperforms the standard expert-choice MoE with 8 mixed-modal experts,
  which achieves 3x overall FLOPs savings (3x for text, 2.8x for image).
  Combining MoMa with mixture-of-depths (MoD) further improves pre-training FLOPs
  savings to 4.2x overall (text: 3.4x, image: 5.3x), although this combination
  hurts performance in causal inference due to increased sensitivity to router
  accuracy. These results demonstrate MoMa's potential to significantly advance
  the efficiency of mixed-modal, early-fusion language model pre-training, paving
  the way for more resource-efficient and capable multimodal AI systems.
paper_ai_summary: MoMa, an architecture for pre-training mixed-modal language models, achieves substantial efficiency savings by using modality-specific experts and mixture-of-depths, though combining these techniques affects causal inference performance.
paper_ai_keywords: modality-aware mixture-of-experts (MoE) early-fusion language models modality-specific groups learned routing semantically informed adaptivity parameter allocation FLOPs savings mixture-of-depths (MoD) causal inference router accuracy