title: mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal
    Large Language Models
summary: Multi-modal Large Language Models (MLLMs) have demonstrated remarkable
  capabilities in executing instructions for a variety of single-image tasks.
  Despite this progress, significant challenges remain in modeling long image
  sequences. In this work, we introduce the versatile multi-modal large language
  model, mPLUG-Owl3, which enhances the capability for long image-sequence
  understanding in scenarios that incorporate retrieved image-text knowledge,
  interleaved image-text, and lengthy videos. Specifically, we propose novel
  hyper attention blocks to efficiently integrate vision and language into a
  common language-guided semantic space, thereby facilitating the processing of
  extended multi-image scenarios. Extensive experimental results suggest that
  mPLUG-Owl3 achieves state-of-the-art performance among models with a similar
  size on single-image, multi-image, and video benchmarks. Moreover, we propose a
  challenging long visual sequence evaluation named Distractor Resistance to
  assess the ability of models to maintain focus amidst distractions. Finally,
  with the proposed architecture, mPLUG-Owl3 demonstrates outstanding performance
  on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to
  the development of more efficient and powerful multimodal large language
  models.
paper_authors._id: 66b9889c430220d6a344681e
paper_authors.name: Jiabo Ye
paper_authors._id: 66b9889c430220d6a344681f
paper_authors.name: Haiyang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T10:03:00.246Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 66b9889c430220d6a3446820
paper_authors.name: Haowei Liu
paper_authors._id: 66b9889c430220d6a3446821
paper_authors.name: Anwen Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T10:04:25.274Z
paper_authors._id: 6584f2f941dbedb146fbb902
paper_authors.avatarUrl: /avatars/9b6e1edecce6c1a2dcf4739be2bfd1b4.svg
paper_authors.fullname: AnwenHu
paper_authors.type: user
paper_authors.user: AnwenHu
paper_authors._id: 66b9889c430220d6a3446822
paper_authors.name: Ming Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T10:04:40.775Z
paper_authors._id: 64771cfdd7cf39f2e9381aa9
paper_authors.avatarUrl: /avatars/48adf00c3b653df02628f80511639e19.svg
paper_authors.fullname: Ming
paper_authors.type: user
paper_authors.user: MingYan123
paper_authors._id: 66b9889c430220d6a3446823
paper_authors.name: Qi Qian
paper_authors._id: 66b9889c430220d6a3446824
paper_authors.name: Ji Zhang
paper_authors._id: 66b9889c430220d6a3446825
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T10:05:40.317Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 66b9889c430220d6a3446826
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T10:05:50.021Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal
    Large Language Models
paper_summary: Multi-modal Large Language Models (MLLMs) have demonstrated remarkable
  capabilities in executing instructions for a variety of single-image tasks.
  Despite this progress, significant challenges remain in modeling long image
  sequences. In this work, we introduce the versatile multi-modal large language
  model, mPLUG-Owl3, which enhances the capability for long image-sequence
  understanding in scenarios that incorporate retrieved image-text knowledge,
  interleaved image-text, and lengthy videos. Specifically, we propose novel
  hyper attention blocks to efficiently integrate vision and language into a
  common language-guided semantic space, thereby facilitating the processing of
  extended multi-image scenarios. Extensive experimental results suggest that
  mPLUG-Owl3 achieves state-of-the-art performance among models with a similar
  size on single-image, multi-image, and video benchmarks. Moreover, we propose a
  challenging long visual sequence evaluation named Distractor Resistance to
  assess the ability of models to maintain focus amidst distractions. Finally,
  with the proposed architecture, mPLUG-Owl3 demonstrates outstanding performance
  on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to
  the development of more efficient and powerful multimodal large language
  models.
paper_ai_keywords: multi-modal large language models (MLLMs) image-text knowledge hyper attention blocks long image-sequence understanding state-of-the-art performance ultra-long visual sequence inputs Distractor Resistance