LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with
  3D-awareness
Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced
their proficiency in 2D visual understanding tasks, enabling them to
effectively process and understand images and videos. However, the development
of LMMs with 3D-awareness for 3D scene understanding has been hindered by the
lack of large-scale 3D vision-language datasets and powerful 3D encoders. In
this paper, we introduce a simple yet effective framework called LLaVA-3D.
Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D
efficiently adapts LLaVA for 3D scene understanding without compromising 2D
understanding capabilities. To achieve this, we employ a simple yet effective
representation, 3D Patch, which connects 2D CLIP patch features with their
corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs
and employing joint 2D and 3D vision-language instruction tuning, we establish
a unified architecture for both 2D image understanding and 3D scene
understanding. Experimental results show that LLaVA-3D converges 3.5x faster
than existing 3D LMMs when trained on 3D vision-language datasets. Moreover,
LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks
but also maintains comparable 2D image understanding and vision-language
conversation capabilities with LLaVA.
66f62da4793e1463e406fda2
Chenming Zhu
admin_assigned
2024-09-27T08:09:16.387Z
6433aba4546e16f17a0f19f6
https://cdn-avatars.huggingface.co/v1/production/uploads/6433aba4546e16f17a0f19f6/3hlNy4_Suy0bO__Qa9amL.jpeg
Chenming Zhu
user
ChaimZhu
66f62da4793e1463e406fda3
Tai Wang
admin_assigned
2024-09-27T08:09:23.888Z
64e6d9d229a548f66aff6e5b
https://cdn-avatars.huggingface.co/v1/production/uploads/64e6d9d229a548f66aff6e5b/yQ9E2TyzM4CfSjMPigcey.jpeg
Tai Wang
user
taiwang
66f62da4793e1463e406fda4
Wenwei Zhang
claimed_verified
2025-02-11T10:04:23.759Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
66f62da4793e1463e406fda5
Jiangmiao Pang
admin_assigned
2024-09-27T08:10:00.337Z
65783ee6ee33d547aecc3ffc
https://cdn-avatars.huggingface.co/v1/production/uploads/65783ee6ee33d547aecc3ffc/lWZX88c-0dCsN-yB9Jhlf.jpeg
Jiangmiao Pang
user
Jiangmiao
66f62da4793e1463e406fda6
Xihui Liu
admin_assigned
2024-09-27T08:10:10.152Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with
  3D-awareness
Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced
their proficiency in 2D visual understanding tasks, enabling them to
effectively process and understand images and videos. However, the development
of LMMs with 3D-awareness for 3D scene understanding has been hindered by the
lack of large-scale 3D vision-language datasets and powerful 3D encoders. In
this paper, we introduce a simple yet effective framework called LLaVA-3D.
Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D
efficiently adapts LLaVA for 3D scene understanding without compromising 2D
understanding capabilities. To achieve this, we employ a simple yet effective
representation, 3D Patch, which connects 2D CLIP patch features with their
corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs
and employing joint 2D and 3D vision-language instruction tuning, we establish
a unified architecture for both 2D image understanding and 3D scene
understanding. Experimental results show that LLaVA-3D converges 3.5x faster
than existing 3D LMMs when trained on 3D vision-language datasets. Moreover,
LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks
but also maintains comparable 2D image understanding and vision-language
conversation capabilities with LLaVA.
LLaVA-3D enhances 2D models with 3D scene understanding by integrating 3D Patch representations and joint 2D/3D instruction tuning, achieving fast convergence and superior performance across tasks.
LMMs LLaVA 3D Patch 2D CLIP patch features joint 2D and 3D vision-language instruction tuning