mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with
  Modality Collaboration
Multi-modal Large Language Models (MLLMs) have demonstrated impressive
instruction abilities across various open-ended tasks. However, previous
methods primarily focus on enhancing multi-modal capabilities. In this work, we
introduce a versatile multi-modal large language model, mPLUG-Owl2, which
effectively leverages modality collaboration to improve performance in both
text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design,
with the language decoder acting as a universal interface for managing
different modalities. Specifically, mPLUG-Owl2 incorporates shared functional
modules to facilitate modality collaboration and introduces a modality-adaptive
module that preserves modality-specific features. Extensive experiments reveal
that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal
tasks and achieving state-of-the-art performances with a single generic model.
Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality
collaboration phenomenon in both pure-text and multi-modal scenarios, setting a
pioneering path in the development of future multi-modal foundation models.
654c436067af1cb876c0f1a2
Qinghao Ye
admin_assigned
2023-11-09T09:25:24.823Z
64530fc01a57e1179c1fe4c0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
QinghaoYe
user
MAGAer13
654c436067af1cb876c0f1a3
Haiyang Xu
claimed_verified
2023-11-09T08:27:23.120Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
654c436067af1cb876c0f1a4
Jiabo Ye
654c436067af1cb876c0f1a5
Ming Yan
claimed_verified
2023-11-09T09:41:01.911Z
64771cfdd7cf39f2e9381aa9
/avatars/48adf00c3b653df02628f80511639e19.svg
Ming
user
MingYan123
654c436067af1cb876c0f1a6
Haowei Liu
654c436067af1cb876c0f1a7
Qi Qian
654c436067af1cb876c0f1a8
Ji Zhang
654c436067af1cb876c0f1a9
Fei Huang
admin_assigned
2023-11-09T09:22:31.132Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
654c436067af1cb876c0f1aa
Jingren Zhou
admin_assigned
2023-11-09T09:22:49.237Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
654cad7fbf910b16a2b6df4e
Anwen HU
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with
  Modality Collaboration
Multi-modal Large Language Models (MLLMs) have demonstrated impressive
instruction abilities across various open-ended tasks. However, previous
methods primarily focus on enhancing multi-modal capabilities. In this work, we
introduce a versatile multi-modal large language model, mPLUG-Owl2, which
effectively leverages modality collaboration to improve performance in both
text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design,
with the language decoder acting as a universal interface for managing
different modalities. Specifically, mPLUG-Owl2 incorporates shared functional
modules to facilitate modality collaboration and introduces a modality-adaptive
module that preserves modality-specific features. Extensive experiments reveal
that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal
tasks and achieving state-of-the-art performances with a single generic model.
Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality
collaboration phenomenon in both pure-text and multi-modal scenarios, setting a
pioneering path in the development of future multi-modal foundation models.
mPLUG-Owl2, a versatile multi-modal large language model, uses a modularized network design to enhance performance in both text and multi-modal tasks through modality collaboration and modality-adaptive modules.
multi-modal large language models MLLMs modality collaboration modularized network design language decoder shared functional modules modality-adaptive module pure-text multi-modal scenarios foundation models