title: LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with
    3D-awareness
summary: Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced
  their proficiency in 2D visual understanding tasks, enabling them to
  effectively process and understand images and videos. However, the development
  of LMMs with 3D-awareness for 3D scene understanding has been hindered by the
  lack of large-scale 3D vision-language datasets and powerful 3D encoders. In
  this paper, we introduce a simple yet effective framework called LLaVA-3D.
  Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D
  efficiently adapts LLaVA for 3D scene understanding without compromising 2D
  understanding capabilities. To achieve this, we employ a simple yet effective
  representation, 3D Patch, which connects 2D CLIP patch features with their
  corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs
  and employing joint 2D and 3D vision-language instruction tuning, we establish
  a unified architecture for both 2D image understanding and 3D scene
  understanding. Experimental results show that LLaVA-3D converges 3.5x faster
  than existing 3D LMMs when trained on 3D vision-language datasets. Moreover,
  LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks
  but also maintains comparable 2D image understanding and vision-language
  conversation capabilities with LLaVA.
paper_authors._id: 66f62da4793e1463e406fda2
paper_authors.name: Chenming Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T08:09:16.387Z
paper_authors._id: 6433aba4546e16f17a0f19f6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6433aba4546e16f17a0f19f6/3hlNy4_Suy0bO__Qa9amL.jpeg
paper_authors.fullname: Chenming Zhu
paper_authors.type: user
paper_authors.user: ChaimZhu
paper_authors._id: 66f62da4793e1463e406fda3
paper_authors.name: Tai Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T08:09:23.888Z
paper_authors._id: 64e6d9d229a548f66aff6e5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e6d9d229a548f66aff6e5b/yQ9E2TyzM4CfSjMPigcey.jpeg
paper_authors.fullname: Tai Wang
paper_authors.type: user
paper_authors.user: taiwang
paper_authors._id: 66f62da4793e1463e406fda4
paper_authors.name: Wenwei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:04:23.759Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 66f62da4793e1463e406fda5
paper_authors.name: Jiangmiao Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T08:10:00.337Z
paper_authors._id: 65783ee6ee33d547aecc3ffc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65783ee6ee33d547aecc3ffc/lWZX88c-0dCsN-yB9Jhlf.jpeg
paper_authors.fullname: Jiangmiao Pang
paper_authors.type: user
paper_authors.user: Jiangmiao
paper_authors._id: 66f62da4793e1463e406fda6
paper_authors.name: Xihui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T08:10:10.152Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_title: LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with
    3D-awareness
paper_summary: Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced
  their proficiency in 2D visual understanding tasks, enabling them to
  effectively process and understand images and videos. However, the development
  of LMMs with 3D-awareness for 3D scene understanding has been hindered by the
  lack of large-scale 3D vision-language datasets and powerful 3D encoders. In
  this paper, we introduce a simple yet effective framework called LLaVA-3D.
  Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D
  efficiently adapts LLaVA for 3D scene understanding without compromising 2D
  understanding capabilities. To achieve this, we employ a simple yet effective
  representation, 3D Patch, which connects 2D CLIP patch features with their
  corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs
  and employing joint 2D and 3D vision-language instruction tuning, we establish
  a unified architecture for both 2D image understanding and 3D scene
  understanding. Experimental results show that LLaVA-3D converges 3.5x faster
  than existing 3D LMMs when trained on 3D vision-language datasets. Moreover,
  LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks
  but also maintains comparable 2D image understanding and vision-language
  conversation capabilities with LLaVA.
paper_ai_summary: LLaVA-3D enhances 2D models with 3D scene understanding by integrating 3D Patch representations and joint 2D/3D instruction tuning, achieving fast convergence and superior performance across tasks.
paper_ai_keywords: LMMs LLaVA 3D Patch 2D CLIP patch features joint 2D and 3D vision-language instruction tuning