title: Task-oriented Sequential Grounding in 3D Scenes
summary: Grounding natural language in physical 3D environments is essential for the
  advancement of embodied artificial intelligence. Current datasets and models
  for 3D visual grounding predominantly focus on identifying and localizing
  objects from static, object-centric descriptions. These approaches do not
  adequately address the dynamic and sequential nature of task-oriented grounding
  necessary for practical applications. In this work, we propose a new task:
  Task-oriented Sequential Grounding in 3D scenes, wherein an agent must follow
  detailed step-by-step instructions to complete daily activities by locating a
  sequence of target objects in indoor scenes. To facilitate this task, we
  introduce SG3D, a large-scale dataset containing 22,346 tasks with 112,236
  steps across 4,895 real-world 3D scenes. The dataset is constructed using a
  combination of RGB-D scans from various 3D scene datasets and an automated task
  generation pipeline, followed by human verification for quality assurance. We
  adapted three state-of-the-art 3D visual grounding models to the sequential
  grounding task and evaluated their performance on SG3D. Our results reveal that
  while these models perform well on traditional benchmarks, they face
  significant challenges with task-oriented sequential grounding, underscoring
  the need for further research in this area.
paper_authors._id: 66b5771021975ba10d9af9c5
paper_authors.name: Zhuofan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:06:49.372Z
paper_authors._id: 663cac5ba4d7c8c9038997d0
paper_authors.avatarUrl: /avatars/f91c5b0f0a1f081b7fa8a37484dc8fbe.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: ZhuofanZhang
paper_authors._id: 66b5771021975ba10d9af9c6
paper_authors.name: Ziyu Zhu
paper_authors._id: 66b5771021975ba10d9af9c7
paper_authors.name: Pengxiang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:47:32.317Z
paper_authors._id: 65d6bd8ebb862d1b367580fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d6bd8ebb862d1b367580fe/Pf5o-Oxi2IZxJPa-ZgEtV.jpeg
paper_authors.fullname: Pengxiang Li
paper_authors.type: user
paper_authors.user: PengxiangLi
paper_authors._id: 66b5771021975ba10d9af9c8
paper_authors.name: Tengyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:08:22.978Z
paper_authors._id: 634379a81bdd3dfa55dcbe82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665366425318-noauth.png
paper_authors.fullname: Tengyu Liu
paper_authors.type: user
paper_authors.user: oxrider
paper_authors._id: 66b5771021975ba10d9af9c9
paper_authors.name: Xiaojian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:08:28.518Z
paper_authors._id: 60dd0e36a15ddd7d2006d2e9
paper_authors.avatarUrl: /avatars/8bd98177a79efbf295be8f6457683297.svg
paper_authors.fullname: Xiaojian Ma
paper_authors.type: user
paper_authors.user: jeasinema
paper_authors._id: 66b5771021975ba10d9af9ca
paper_authors.name: Yixin Chen
paper_authors._id: 66b5771021975ba10d9af9cb
paper_authors.name: Baoxiong Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:08:55.418Z
paper_authors._id: 6304b389bad6ce7fc02691d5
paper_authors.avatarUrl: /avatars/a762ca59624ce409650165f36b973488.svg
paper_authors.fullname: Baoxiong Jia
paper_authors.type: user
paper_authors.user: BuzzBeater
paper_authors._id: 66b5771021975ba10d9af9cc
paper_authors.name: Siyuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:09:11.202Z
paper_authors._id: 63c7a33121bd95f80ed74652
paper_authors.avatarUrl: /avatars/7dd59afea785a2bff0ec2b757abd474e.svg
paper_authors.fullname: Siyuan Huang
paper_authors.type: user
paper_authors.user: thuhsy
paper_authors._id: 66b5771021975ba10d9af9cd
paper_authors.name: Qing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T14:52:35.788Z
paper_authors._id: 6455f8c2c8f569b995d603c9
paper_authors.avatarUrl: /avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
paper_authors.fullname: Qing Li
paper_authors.type: user
paper_authors.user: li-qing
paper_title: Task-oriented Sequential Grounding in 3D Scenes
paper_summary: Grounding natural language in physical 3D environments is essential for the
  advancement of embodied artificial intelligence. Current datasets and models
  for 3D visual grounding predominantly focus on identifying and localizing
  objects from static, object-centric descriptions. These approaches do not
  adequately address the dynamic and sequential nature of task-oriented grounding
  necessary for practical applications. In this work, we propose a new task:
  Task-oriented Sequential Grounding in 3D scenes, wherein an agent must follow
  detailed step-by-step instructions to complete daily activities by locating a
  sequence of target objects in indoor scenes. To facilitate this task, we
  introduce SG3D, a large-scale dataset containing 22,346 tasks with 112,236
  steps across 4,895 real-world 3D scenes. The dataset is constructed using a
  combination of RGB-D scans from various 3D scene datasets and an automated task
  generation pipeline, followed by human verification for quality assurance. We
  adapted three state-of-the-art 3D visual grounding models to the sequential
  grounding task and evaluated their performance on SG3D. Our results reveal that
  while these models perform well on traditional benchmarks, they face
  significant challenges with task-oriented sequential grounding, underscoring
  the need for further research in this area.
paper_ai_summary: A new dataset and task for sequential task-oriented grounding in 3D environments are introduced to advance embodied AI, showcasing challenges for current models in dynamic, instruction-following tasks.
paper_ai_keywords: 3D visual grounding task-oriented sequential grounding SG3D RGB-D scans automated task generation human verification state-of-the-art 3D visual grounding models