LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large
  Multimodal Models
Visual instruction tuning has made considerable strides in enhancing the
capabilities of Large Multimodal Models (LMMs). However, existing open LMMs
largely focus on single-image tasks, their applications to multi-image
scenarios remains less explored. Additionally, prior LMM research separately
tackles different scenarios, leaving it impossible to generalize cross
scenarios with new emerging capabilities. To this end, we introduce
LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame
(video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To
enable these capabilities, we regard the interleaved data format as a general
template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4
primary domains with 14 tasks and 41 datasets. We also curate the
LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance
of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading
results in multi-image, video, and 3D benchmarks, while maintaining the
performance of single-image tasks. Besides, our model also exhibits several
emerging capabilities, e.g., transferring tasks across different settings and
modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT
668f4cd8c0af06ce53c13a26
Feng Li
668f4cd8c0af06ce53c13a27
Renrui Zhang
claimed_verified
2024-07-11T08:25:47.769Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
668f4cd8c0af06ce53c13a28
Hao Zhang
claimed_verified
2024-07-11T13:45:41.888Z
642e5a066748dd4f8eebcd16
https://cdn-avatars.huggingface.co/v1/production/uploads/642e5a066748dd4f8eebcd16/7ZYI0_8Z8cfsiG6r7_Y-p.jpeg
Hao Zhang
user
Haozhangcx
668f4cd8c0af06ce53c13a29
Yuanhan Zhang
admin_assigned
2024-07-11T08:56:09.879Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
668f4cd8c0af06ce53c13a2a
Bo Li
admin_assigned
2024-07-11T08:57:01.301Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
668f4cd8c0af06ce53c13a2b
Wei Li
668f4cd8c0af06ce53c13a2c
Zejun Ma
668f4cd8c0af06ce53c13a2d
Chunyuan Li
admin_assigned
2024-07-11T08:56:18.129Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large
  Multimodal Models
Visual instruction tuning has made considerable strides in enhancing the
capabilities of Large Multimodal Models (LMMs). However, existing open LMMs
largely focus on single-image tasks, their applications to multi-image
scenarios remains less explored. Additionally, prior LMM research separately
tackles different scenarios, leaving it impossible to generalize cross
scenarios with new emerging capabilities. To this end, we introduce
LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame
(video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To
enable these capabilities, we regard the interleaved data format as a general
template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4
primary domains with 14 tasks and 41 datasets. We also curate the
LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance
of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading
results in multi-image, video, and 3D benchmarks, while maintaining the
performance of single-image tasks. Besides, our model also exhibits several
emerging capabilities, e.g., transferring tasks across different settings and
modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT
LLaVA-NeXT-Interleave addresses multi-image, multi-frame, multi-view, and multi-patch scenarios in LMMs using the M4-Instruct dataset and LLaVA-Interleave Bench to achieve leading results while maintaining single-image performance.
Large Multimodal Models LLMs Multi-image Multi-frame Multi-view Multi-patch interleaved data format M4-Instruct dataset LLaVA-Interleave Bench