title: LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large
    Multimodal Models
summary: Visual instruction tuning has made considerable strides in enhancing the
  capabilities of Large Multimodal Models (LMMs). However, existing open LMMs
  largely focus on single-image tasks, their applications to multi-image
  scenarios remains less explored. Additionally, prior LMM research separately
  tackles different scenarios, leaving it impossible to generalize cross
  scenarios with new emerging capabilities. To this end, we introduce
  LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame
  (video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To
  enable these capabilities, we regard the interleaved data format as a general
  template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4
  primary domains with 14 tasks and 41 datasets. We also curate the
  LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance
  of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading
  results in multi-image, video, and 3D benchmarks, while maintaining the
  performance of single-image tasks. Besides, our model also exhibits several
  emerging capabilities, e.g., transferring tasks across different settings and
  modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT
paper_authors._id: 668f4cd8c0af06ce53c13a26
paper_authors.name: Feng Li
paper_authors._id: 668f4cd8c0af06ce53c13a27
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T08:25:47.769Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 668f4cd8c0af06ce53c13a28
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T13:45:41.888Z
paper_authors._id: 642e5a066748dd4f8eebcd16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e5a066748dd4f8eebcd16/7ZYI0_8Z8cfsiG6r7_Y-p.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: Haozhangcx
paper_authors._id: 668f4cd8c0af06ce53c13a29
paper_authors.name: Yuanhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T08:56:09.879Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 668f4cd8c0af06ce53c13a2a
paper_authors.name: Bo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T08:57:01.301Z
paper_authors._id: 62d3f7d84b0933c48f3cdd9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: luodian
paper_authors._id: 668f4cd8c0af06ce53c13a2b
paper_authors.name: Wei Li
paper_authors._id: 668f4cd8c0af06ce53c13a2c
paper_authors.name: Zejun Ma
paper_authors._id: 668f4cd8c0af06ce53c13a2d
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T08:56:18.129Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_title: LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large
    Multimodal Models
paper_summary: Visual instruction tuning has made considerable strides in enhancing the
  capabilities of Large Multimodal Models (LMMs). However, existing open LMMs
  largely focus on single-image tasks, their applications to multi-image
  scenarios remains less explored. Additionally, prior LMM research separately
  tackles different scenarios, leaving it impossible to generalize cross
  scenarios with new emerging capabilities. To this end, we introduce
  LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame
  (video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To
  enable these capabilities, we regard the interleaved data format as a general
  template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4
  primary domains with 14 tasks and 41 datasets. We also curate the
  LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance
  of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading
  results in multi-image, video, and 3D benchmarks, while maintaining the
  performance of single-image tasks. Besides, our model also exhibits several
  emerging capabilities, e.g., transferring tasks across different settings and
  modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT
paper_ai_summary: LLaVA-NeXT-Interleave addresses multi-image, multi-frame, multi-view, and multi-patch scenarios in LMMs using the M4-Instruct dataset and LLaVA-Interleave Bench to achieve leading results while maintaining single-image performance.
paper_ai_keywords: Large Multimodal Models LLMs Multi-image Multi-frame Multi-view Multi-patch interleaved data format M4-Instruct dataset LLaVA-Interleave Bench