MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware
  Experts
We introduce MoMa, a novel modality-aware mixture-of-experts (MoE)
architecture designed for pre-training mixed-modal, early-fusion language
models. MoMa processes images and text in arbitrary sequences by dividing
expert modules into modality-specific groups. These groups exclusively process
designated tokens while employing learned routing within each group to maintain
semantically informed adaptivity. Our empirical results reveal substantial
pre-training efficiency gains through this modality-specific parameter
allocation. Under a 1-trillion-token training budget, the MoMa 1.4B model,
featuring 4 text experts and 4 image experts, achieves impressive FLOPs
savings: 3.7x overall, with 2.6x for text and 5.2x for image processing
compared to a compute-equivalent dense baseline, measured by pre-training loss.
This outperforms the standard expert-choice MoE with 8 mixed-modal experts,
which achieves 3x overall FLOPs savings (3x for text, 2.8x for image).
Combining MoMa with mixture-of-depths (MoD) further improves pre-training FLOPs
savings to 4.2x overall (text: 3.4x, image: 5.3x), although this combination
hurts performance in causal inference due to increased sensitivity to router
accuracy. These results demonstrate MoMa's potential to significantly advance
the efficiency of mixed-modal, early-fusion language model pre-training, paving
the way for more resource-efficient and capable multimodal AI systems.
66aaeffdadfd339ee6e4a0d8
Xi Victoria Lin
admin_assigned
2024-08-02T15:06:16.361Z
61df349e34b589fbaf2e93c4
/avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
Victoria Lin
user
VictoriaLinML
66aaeffdadfd339ee6e4a0d9
Akshat Shrivastava
extracted_pending
2024-08-01T02:16:30.719Z
64b6d105f616b195a39f462a
/avatars/f33805c3e42e476a135c3cf869cc423a.svg
Akshat
user
akshats07
66aaeffdadfd339ee6e4a0da
Liang Luo
admin_assigned
2024-08-02T15:06:36.409Z
65e8be4b45ae4e903cee573a
/avatars/546e503dc13a1c081c6b139f551b5b00.svg
Liang Luo
user
liangluo
66aaeffdadfd339ee6e4a0db
Srinivasan Iyer
66aaeffdadfd339ee6e4a0dc
Mike Lewis
admin_assigned
2024-08-02T15:07:10.452Z
5e67eda71009063689407484
/avatars/6d7688faba888d463584eaa02ad011e3.svg
Mike Lewis
user
mikelewis0
66aaeffdadfd339ee6e4a0dd
Gargi Gosh
66aaeffdadfd339ee6e4a0de
Luke Zettlemoyer
66aaeffdadfd339ee6e4a0df
Armen Aghajanyan
extracted_pending
2024-10-24T21:25:49.162Z
671abaf6e0813cc6983103c6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/SLytMjnVS452F9qhF_GFQ.png
Armen Aghajanyan
user
ArmenAgh
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware
  Experts
We introduce MoMa, a novel modality-aware mixture-of-experts (MoE)
architecture designed for pre-training mixed-modal, early-fusion language
models. MoMa processes images and text in arbitrary sequences by dividing
expert modules into modality-specific groups. These groups exclusively process
designated tokens while employing learned routing within each group to maintain
semantically informed adaptivity. Our empirical results reveal substantial
pre-training efficiency gains through this modality-specific parameter
allocation. Under a 1-trillion-token training budget, the MoMa 1.4B model,
featuring 4 text experts and 4 image experts, achieves impressive FLOPs
savings: 3.7x overall, with 2.6x for text and 5.2x for image processing
compared to a compute-equivalent dense baseline, measured by pre-training loss.
This outperforms the standard expert-choice MoE with 8 mixed-modal experts,
which achieves 3x overall FLOPs savings (3x for text, 2.8x for image).
Combining MoMa with mixture-of-depths (MoD) further improves pre-training FLOPs
savings to 4.2x overall (text: 3.4x, image: 5.3x), although this combination
hurts performance in causal inference due to increased sensitivity to router
accuracy. These results demonstrate MoMa's potential to significantly advance
the efficiency of mixed-modal, early-fusion language model pre-training, paving
the way for more resource-efficient and capable multimodal AI systems.
MoMa, an architecture for pre-training mixed-modal language models, achieves substantial efficiency savings by using modality-specific experts and mixture-of-depths, though combining these techniques affects causal inference performance.
modality-aware mixture-of-experts (MoE) early-fusion language models modality-specific groups learned routing semantically informed adaptivity parameter allocation FLOPs savings mixture-of-depths (MoD) causal inference router accuracy