title: Scaling Laws for Native Multimodal Models Scaling Laws for Native
    Multimodal Models
summary: Building general-purpose models that can effectively perceive the world
  through multimodal signals has been a long-standing goal. Current approaches
  involve integrating separately pre-trained components, such as connecting
  vision encoders to LLMs and continuing multimodal training. While such
  approaches exhibit remarkable sample efficiency, it remains an open question
  whether such late-fusion architectures are inherently superior. In this work,
  we revisit the architectural design of native multimodal models (NMMs)--those
  trained from the ground up on all modalities--and conduct an extensive scaling
  laws study, spanning 457 trained models with different architectures and
  training mixtures. Our investigation reveals no inherent advantage to
  late-fusion architectures over early-fusion ones, which do not rely on image
  encoders. On the contrary, early-fusion exhibits stronger performance at lower
  parameter counts, is more efficient to train, and is easier to deploy.
  Motivated by the strong performance of the early-fusion architectures, we show
  that incorporating Mixture of Experts (MoEs) allows for models that learn
  modality-specific weights, significantly enhancing performance.
paper_authors._id: 67f8e4acc7b30121f9c054d1
paper_authors.name: Mustafa Shukor
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:14:19.280Z
paper_authors._id: 62bdeedd01dc22b4d22a371e
paper_authors.avatarUrl: /avatars/6adc904fb1e08661d293a966270afabb.svg
paper_authors.fullname: Mustafa Shukor
paper_authors.type: user
paper_authors.user: mshukor
paper_authors._id: 67f8e4acc7b30121f9c054d2
paper_authors.name: Enrico Fini
paper_authors._id: 67f8e4acc7b30121f9c054d3
paper_authors.name: Victor Guilherme Turrisi da Costa
paper_authors._id: 67f8e4acc7b30121f9c054d4
paper_authors.name: Matthieu Cord
paper_authors._id: 67f8e4acc7b30121f9c054d5
paper_authors.name: Joshua Susskind
paper_authors._id: 67f8e4acc7b30121f9c054d6
paper_authors.name: Alaaeldin El-Nouby
paper_title: Scaling Laws for Native Multimodal Models Scaling Laws for Native
    Multimodal Models
paper_summary: Building general-purpose models that can effectively perceive the world
  through multimodal signals has been a long-standing goal. Current approaches
  involve integrating separately pre-trained components, such as connecting
  vision encoders to LLMs and continuing multimodal training. While such
  approaches exhibit remarkable sample efficiency, it remains an open question
  whether such late-fusion architectures are inherently superior. In this work,
  we revisit the architectural design of native multimodal models (NMMs)--those
  trained from the ground up on all modalities--and conduct an extensive scaling
  laws study, spanning 457 trained models with different architectures and
  training mixtures. Our investigation reveals no inherent advantage to
  late-fusion architectures over early-fusion ones, which do not rely on image
  encoders. On the contrary, early-fusion exhibits stronger performance at lower
  parameter counts, is more efficient to train, and is easier to deploy.
  Motivated by the strong performance of the early-fusion architectures, we show
  that incorporating Mixture of Experts (MoEs) allows for models that learn
  modality-specific weights, significantly enhancing performance.
paper_ai_summary: Early-fusion architectures, which do not rely on image encoders, outperform late-fusion models in native multimodal architectures, especially when enhanced with Mixture of Experts (MoEs).
paper_ai_keywords: native multimodal models early-fusion late-fusion Mixture of Experts