mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal
  Large Language Models
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable
capabilities in executing instructions for a variety of single-image tasks.
Despite this progress, significant challenges remain in modeling long image
sequences. In this work, we introduce the versatile multi-modal large language
model, mPLUG-Owl3, which enhances the capability for long image-sequence
understanding in scenarios that incorporate retrieved image-text knowledge,
interleaved image-text, and lengthy videos. Specifically, we propose novel
hyper attention blocks to efficiently integrate vision and language into a
common language-guided semantic space, thereby facilitating the processing of
extended multi-image scenarios. Extensive experimental results suggest that
mPLUG-Owl3 achieves state-of-the-art performance among models with a similar
size on single-image, multi-image, and video benchmarks. Moreover, we propose a
challenging long visual sequence evaluation named Distractor Resistance to
assess the ability of models to maintain focus amidst distractions. Finally,
with the proposed architecture, mPLUG-Owl3 demonstrates outstanding performance
on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to
the development of more efficient and powerful multimodal large language
models.
66b9889c430220d6a344681e
Jiabo Ye
66b9889c430220d6a344681f
Haiyang Xu
admin_assigned
2024-08-13T10:03:00.246Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
66b9889c430220d6a3446820
Haowei Liu
66b9889c430220d6a3446821
Anwen Hu
admin_assigned
2024-08-13T10:04:25.274Z
6584f2f941dbedb146fbb902
/avatars/9b6e1edecce6c1a2dcf4739be2bfd1b4.svg
AnwenHu
user
AnwenHu
66b9889c430220d6a3446822
Ming Yan
admin_assigned
2024-08-13T10:04:40.775Z
64771cfdd7cf39f2e9381aa9
/avatars/48adf00c3b653df02628f80511639e19.svg
Ming
user
MingYan123
66b9889c430220d6a3446823
Qi Qian
66b9889c430220d6a3446824
Ji Zhang
66b9889c430220d6a3446825
Fei Huang
admin_assigned
2024-08-13T10:05:40.317Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
66b9889c430220d6a3446826
Jingren Zhou
admin_assigned
2024-08-13T10:05:50.021Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal
  Large Language Models
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable
capabilities in executing instructions for a variety of single-image tasks.
Despite this progress, significant challenges remain in modeling long image
sequences. In this work, we introduce the versatile multi-modal large language
model, mPLUG-Owl3, which enhances the capability for long image-sequence
understanding in scenarios that incorporate retrieved image-text knowledge,
interleaved image-text, and lengthy videos. Specifically, we propose novel
hyper attention blocks to efficiently integrate vision and language into a
common language-guided semantic space, thereby facilitating the processing of
extended multi-image scenarios. Extensive experimental results suggest that
mPLUG-Owl3 achieves state-of-the-art performance among models with a similar
size on single-image, multi-image, and video benchmarks. Moreover, we propose a
challenging long visual sequence evaluation named Distractor Resistance to
assess the ability of models to maintain focus amidst distractions. Finally,
with the proposed architecture, mPLUG-Owl3 demonstrates outstanding performance
on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to
the development of more efficient and powerful multimodal large language
models.
mPLUG-Owl3, a versatile multi-modal large language model, introduces hyper attention blocks for improved integration of vision and language, achieving state-of-the-art performance on single-image, multi-image, and video tasks, particularly in scenarios with long visual sequences.
multi-modal large language models MLLMs mPLUG-Owl3 hyper attention blocks vision and language integration semantic space Distractor Resistance long visual sequence inputs