title: Explore the Limits of Omni-modal Pretraining at Scale
summary: We propose to build omni-modal intelligence, which is capable of
  understanding any modality and learning universal representations. In specific,
  we propose a scalable pretraining paradigm, named Multimodal Context (MiCo),
  which can scale up the numbers of modalities and amount of data, together with
  the model parameters, in the pretraining process. With MiCo, the pretrained
  models show significant emergent abilities in multimodal learning, which are
  evaluated on the following tasks: i) single-modality perception benchmarks of
  10 different modalities, ii) 25 cross-modality understanding tasks of
  retrieval, question-answering, captioning, and iii) 18 multimodal large
  language model benchmarks. Our models establish 37 new records for
  state-of-the-art performance. We hope that our research could contribute to the
  development of omni-modal intelligence. Code and Models are at
  https://github.com/invictus717/MiCo
paper_authors._id: 666bada43637dcab935c1fce
paper_authors.name: Yiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:42:10.249Z
paper_authors._id: 63176933b58b0184630d2c74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
paper_authors.fullname: Yiyuan Zhang
paper_authors.type: user
paper_authors.user: Yiyuan
paper_authors._id: 666bada43637dcab935c1fcf
paper_authors.name: Handong Li
paper_authors._id: 666bada43637dcab935c1fd0
paper_authors.name: Jing Liu
paper_authors._id: 666bada43637dcab935c1fd1
paper_authors.name: Xiangyu Yue
paper_title: Explore the Limits of Omni-modal Pretraining at Scale
paper_summary: We propose to build omni-modal intelligence, which is capable of
  understanding any modality and learning universal representations. In specific,
  we propose a scalable pretraining paradigm, named Multimodal Context (MiCo),
  which can scale up the numbers of modalities and amount of data, together with
  the model parameters, in the pretraining process. With MiCo, the pretrained
  models show significant emergent abilities in multimodal learning, which are
  evaluated on the following tasks: i) single-modality perception benchmarks of
  10 different modalities, ii) 25 cross-modality understanding tasks of
  retrieval, question-answering, captioning, and iii) 18 multimodal large
  language model benchmarks. Our models establish 37 new records for
  state-of-the-art performance. We hope that our research could contribute to the
  development of omni-modal intelligence. Code and Models are at
  https://github.com/invictus717/MiCo
paper_ai_summary: Multimodal Context (MiCo) is a scalable pretraining paradigm that enhances multimodal intelligence across various tasks and modalities, achieving state-of-the-art performance.
paper_ai_keywords: Multimodal Context MiCo multimodal intelligence scalable pretraining single-modality perception cross-modality understanding multimodal large language model