title: SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene
    Understanding
summary: 3D vision-language grounding, which focuses on aligning language with the 3D
  physical environment, stands as a cornerstone in the development of embodied
  agents. In comparison to recent advancements in the 2D domain, grounding
  language in 3D scenes faces several significant challenges: (i) the inherent
  complexity of 3D scenes due to the diverse object configurations, their rich
  attributes, and intricate relationships; (ii) the scarcity of paired 3D
  vision-language data to support grounded learning; and (iii) the absence of a
  unified learning framework to distill knowledge from grounded 3D data. In this
  work, we aim to address these three major challenges in 3D vision-language by
  examining the potential of systematically upscaling 3D vision-language learning
  in indoor environments. We introduce the first million-scale 3D vision-language
  dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising
  2.5M vision-language pairs derived from both human annotations and our scalable
  scene-graph-based generation approach. We demonstrate that this scaling allows
  for a unified pre-training framework, Grounded Pre-training for Scenes (GPS),
  for 3D vision-language learning. Through extensive experiments, we showcase the
  effectiveness of GPS by achieving state-of-the-art performance on all existing
  3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is
  unveiled through zero-shot transfer experiments in the challenging 3D
  vision-language tasks. Project website: https://scene-verse.github.io .
paper_authors._id: 65a8ab10f45ee5e5b56d5ca5
paper_authors.name: Baoxiong Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T08:41:38.451Z
paper_authors._id: 6304b389bad6ce7fc02691d5
paper_authors.avatarUrl: /avatars/a762ca59624ce409650165f36b973488.svg
paper_authors.fullname: Baoxiong Jia
paper_authors.type: user
paper_authors.user: BuzzBeater
paper_authors._id: 65a8ab10f45ee5e5b56d5ca6
paper_authors.name: Yixin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T08:42:00.150Z
paper_authors._id: 64c396def6fe448b1ad553d6
paper_authors.avatarUrl: /avatars/b2ce4739f42dc00ee974fff7ee1cb301.svg
paper_authors.fullname: Yixin Chen
paper_authors.type: user
paper_authors.user: YixinChen
paper_authors._id: 65a8ab10f45ee5e5b56d5ca7
paper_authors.name: Huangyue Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T08:43:13.830Z
paper_authors._id: 65a8cbd4669921943c3c0963
paper_authors.avatarUrl: /avatars/c85e1253b8e3ea71eef677d5cd356a55.svg
paper_authors.fullname: Yu
paper_authors.type: user
paper_authors.user: Huangyueeeee
paper_authors._id: 65a8ab10f45ee5e5b56d5ca8
paper_authors.name: Yan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-18T08:19:33.225Z
paper_authors._id: 65a8cc0516e8e332e7515d7d
paper_authors.avatarUrl: /avatars/4a84d28730776962242db6129cf0c4c1.svg
paper_authors.fullname: Yan Wang
paper_authors.type: user
paper_authors.user: Yannnnnnnn
paper_authors._id: 65a8ab10f45ee5e5b56d5ca9
paper_authors.name: Xuesong Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T08:43:19.971Z
paper_authors._id: 64620ca8a489ecb6b6b5b4e4
paper_authors.avatarUrl: /avatars/9bf8060cb7bc59c0369ddf14c50aa480.svg
paper_authors.fullname: Xuesong Niu
paper_authors.type: user
paper_authors.user: nxsEdson
paper_authors._id: 65a8ab10f45ee5e5b56d5caa
paper_authors.name: Tengyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T08:43:27.043Z
paper_authors._id: 634379a81bdd3dfa55dcbe82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665366425318-noauth.png
paper_authors.fullname: Tengyu Liu
paper_authors.type: user
paper_authors.user: oxrider
paper_authors._id: 65a8ab10f45ee5e5b56d5cab
paper_authors.name: Qing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-03T08:26:27.371Z
paper_authors._id: 6455f8c2c8f569b995d603c9
paper_authors.avatarUrl: /avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
paper_authors.fullname: Qing Li
paper_authors.type: user
paper_authors.user: li-qing
paper_authors._id: 65a8ab10f45ee5e5b56d5cac
paper_authors.name: Siyuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T08:44:13.905Z
paper_authors._id: 63c7a33121bd95f80ed74652
paper_authors.avatarUrl: /avatars/7dd59afea785a2bff0ec2b757abd474e.svg
paper_authors.fullname: Siyuan Huang
paper_authors.type: user
paper_authors.user: thuhsy
paper_title: SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene
    Understanding
paper_summary: 3D vision-language grounding, which focuses on aligning language with the 3D
  physical environment, stands as a cornerstone in the development of embodied
  agents. In comparison to recent advancements in the 2D domain, grounding
  language in 3D scenes faces several significant challenges: (i) the inherent
  complexity of 3D scenes due to the diverse object configurations, their rich
  attributes, and intricate relationships; (ii) the scarcity of paired 3D
  vision-language data to support grounded learning; and (iii) the absence of a
  unified learning framework to distill knowledge from grounded 3D data. In this
  work, we aim to address these three major challenges in 3D vision-language by
  examining the potential of systematically upscaling 3D vision-language learning
  in indoor environments. We introduce the first million-scale 3D vision-language
  dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising
  2.5M vision-language pairs derived from both human annotations and our scalable
  scene-graph-based generation approach. We demonstrate that this scaling allows
  for a unified pre-training framework, Grounded Pre-training for Scenes (GPS),
  for 3D vision-language learning. Through extensive experiments, we showcase the
  effectiveness of GPS by achieving state-of-the-art performance on all existing
  3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is
  unveiled through zero-shot transfer experiments in the challenging 3D
  vision-language tasks. Project website: https://scene-verse.github.io .
paper_ai_summary: A large-scale 3D vision-language dataset and pre-training framework achieve state-of-the-art results in 3D visual grounding tasks.
paper_ai_keywords: 3D vision-language grounding grounded learning 3D scenes object configurations rich attributes intricate relationships paired 3D vision-language data unified learning framework SceneVerse vision-language pairs scene-graph-based generation Grounded Pre-training for Scenes GPS 3D visual grounding benchmarks zero-shot transfer