title: mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with
    Modality Collaboration
summary: Multi-modal Large Language Models (MLLMs) have demonstrated impressive
  instruction abilities across various open-ended tasks. However, previous
  methods primarily focus on enhancing multi-modal capabilities. In this work, we
  introduce a versatile multi-modal large language model, mPLUG-Owl2, which
  effectively leverages modality collaboration to improve performance in both
  text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design,
  with the language decoder acting as a universal interface for managing
  different modalities. Specifically, mPLUG-Owl2 incorporates shared functional
  modules to facilitate modality collaboration and introduces a modality-adaptive
  module that preserves modality-specific features. Extensive experiments reveal
  that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal
  tasks and achieving state-of-the-art performances with a single generic model.
  Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality
  collaboration phenomenon in both pure-text and multi-modal scenarios, setting a
  pioneering path in the development of future multi-modal foundation models.
paper_authors._id: 654c436067af1cb876c0f1a2
paper_authors.name: Qinghao Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:25:24.823Z
paper_authors._id: 64530fc01a57e1179c1fe4c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
paper_authors.fullname: QinghaoYe
paper_authors.type: user
paper_authors.user: MAGAer13
paper_authors._id: 654c436067af1cb876c0f1a3
paper_authors.name: Haiyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-09T08:27:23.120Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 654c436067af1cb876c0f1a4
paper_authors.name: Jiabo Ye
paper_authors._id: 654c436067af1cb876c0f1a5
paper_authors.name: Ming Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-09T09:41:01.911Z
paper_authors._id: 64771cfdd7cf39f2e9381aa9
paper_authors.avatarUrl: /avatars/48adf00c3b653df02628f80511639e19.svg
paper_authors.fullname: Ming
paper_authors.type: user
paper_authors.user: MingYan123
paper_authors._id: 654c436067af1cb876c0f1a6
paper_authors.name: Haowei Liu
paper_authors._id: 654c436067af1cb876c0f1a7
paper_authors.name: Qi Qian
paper_authors._id: 654c436067af1cb876c0f1a8
paper_authors.name: Ji Zhang
paper_authors._id: 654c436067af1cb876c0f1a9
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:22:31.132Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 654c436067af1cb876c0f1aa
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:22:49.237Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_authors._id: 654cad7fbf910b16a2b6df4e
paper_authors.name: Anwen HU
paper_title: mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with
    Modality Collaboration
paper_summary: Multi-modal Large Language Models (MLLMs) have demonstrated impressive
  instruction abilities across various open-ended tasks. However, previous
  methods primarily focus on enhancing multi-modal capabilities. In this work, we
  introduce a versatile multi-modal large language model, mPLUG-Owl2, which
  effectively leverages modality collaboration to improve performance in both
  text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design,
  with the language decoder acting as a universal interface for managing
  different modalities. Specifically, mPLUG-Owl2 incorporates shared functional
  modules to facilitate modality collaboration and introduces a modality-adaptive
  module that preserves modality-specific features. Extensive experiments reveal
  that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal
  tasks and achieving state-of-the-art performances with a single generic model.
  Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality
  collaboration phenomenon in both pure-text and multi-modal scenarios, setting a
  pioneering path in the development of future multi-modal foundation models.
paper_ai_summary: mPLUG-Owl2, a versatile multi-modal large language model, uses a modularized network design to enhance performance in both text and multi-modal tasks through modality collaboration and modality-adaptive modules.
paper_ai_keywords: multi-modal large language models MLLMs modality collaboration modularized network design language decoder shared functional modules modality-adaptive module pure-text multi-modal scenarios foundation models