title: Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene
    Understanding
summary: The rapid advancement of Multimodal Large Language Models (MLLMs) has
  significantly impacted various multimodal tasks. However, these models face
  challenges in tasks that require spatial understanding within 3D environments.
  Efforts to enhance MLLMs, such as incorporating point cloud features, have been
  made, yet a considerable gap remains between the models' learned
  representations and the inherent complexity of 3D scenes. This discrepancy
  largely stems from the training of MLLMs on predominantly 2D data, which
  restricts their effectiveness in comprehending 3D spaces. To address this
  issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM,
  for 3D scene understanding. By treating 3D scenes as dynamic videos and
  incorporating 3D position encoding into these representations, our Video-3D LLM
  aligns video representations with real-world spatial contexts more accurately.
  Additionally, we have implemented a maximum coverage sampling technique to
  optimize the balance between computational costs and performance efficiency.
  Extensive experiments demonstrate that our model achieves state-of-the-art
  performance on several 3D scene understanding benchmarks, including ScanRefer,
  Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
paper_authors._id: 675112f1d5d2963818c49068
paper_authors.name: Duo Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:18:28.356Z
paper_authors._id: 646e2fcaf813cfe153f1af6c
paper_authors.avatarUrl: /avatars/2f87d0e5c071000990da29cd744bc03d.svg
paper_authors.fullname: Duo Zheng
paper_authors.type: user
paper_authors.user: zd11024
paper_authors._id: 675112f1d5d2963818c49069
paper_authors.name: Shijia Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:41:36.198Z
paper_authors._id: 65204fdf377bffdc59602a8d
paper_authors.avatarUrl: /avatars/02d200f856f03696b183fd0b59e9f909.svg
paper_authors.fullname: Shijia Huang
paper_authors.type: user
paper_authors.user: slvjul
paper_authors._id: 675112f1d5d2963818c4906a
paper_authors.name: Liwei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:41:42.748Z
paper_authors._id: 6497272b6401266eb6752b89
paper_authors.avatarUrl: /avatars/d94a011660f4cad5344756027840dc30.svg
paper_authors.fullname: Liwei Wang
paper_authors.type: user
paper_authors.user: asdfg80
paper_title: Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene
    Understanding
paper_summary: The rapid advancement of Multimodal Large Language Models (MLLMs) has
  significantly impacted various multimodal tasks. However, these models face
  challenges in tasks that require spatial understanding within 3D environments.
  Efforts to enhance MLLMs, such as incorporating point cloud features, have been
  made, yet a considerable gap remains between the models' learned
  representations and the inherent complexity of 3D scenes. This discrepancy
  largely stems from the training of MLLMs on predominantly 2D data, which
  restricts their effectiveness in comprehending 3D spaces. To address this
  issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM,
  for 3D scene understanding. By treating 3D scenes as dynamic videos and
  incorporating 3D position encoding into these representations, our Video-3D LLM
  aligns video representations with real-world spatial contexts more accurately.
  Additionally, we have implemented a maximum coverage sampling technique to
  optimize the balance between computational costs and performance efficiency.
  Extensive experiments demonstrate that our model achieves state-of-the-art
  performance on several 3D scene understanding benchmarks, including ScanRefer,
  Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) spatial understanding 3D environments point cloud features 3D scenes 2D data Video-3D LLM dynamic videos 3D position encoding maximum coverage sampling technique ScanRefer Multi3DRefer Scan2Cap ScanQA SQA3D