title: LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual
    Preferences
summary: Research on 3D Vision-Language Models (3D-VLMs) is gaining increasing
  attention, which is crucial for developing embodied AI within 3D scenes, such
  as visual navigation and embodied question answering. Due to the high density
  of visual features, especially in large 3D scenes, accurately locating
  task-relevant visual information is challenging. Existing works attempt to
  segment all objects and consider their features as scene representations.
  However, these task-agnostic object features include much redundant information
  and missing details for the task-relevant area. To tackle these problems, we
  propose LSceneLLM, an adaptive framework that automatically identifies
  task-relevant areas by leveraging LLM's visual preference for different tasks,
  followed by a plug-and-play scene magnifier module to capture fine-grained
  details in focused areas. Specifically, a dense token selector examines the
  attention map of LLM to identify visual preferences for the instruction input.
  It then magnifies fine-grained details of the focusing area. An adaptive
  self-attention module is leveraged to fuse the coarse-grained and selected
  fine-grained visual information. To comprehensively evaluate the large scene
  understanding ability of 3D-VLMs, we further introduce a cross-room
  understanding benchmark, XR-Scene, which contains a series of large scene
  understanding tasks including XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption.
  Experiments show that our method surpasses existing methods on both large scene
  understanding and existing scene understanding benchmarks. Plunging our scene
  magnifier module into the existing 3D-VLMs also brings significant improvement.
paper_authors._id: 674e7370881f85b67741483e
paper_authors.name: Hongyan Zhi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:15:16.287Z
paper_authors._id: 674b2406591d7232820252cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bhDlUVkFQ66yt3BEVs6WU.png
paper_authors.fullname: Hongyan Zhi
paper_authors.type: user
paper_authors.user: Hoyard
paper_authors._id: 674e7370881f85b67741483f
paper_authors.name: Peihao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:03:06.868Z
paper_authors._id: 64bfe895c874dda773303b44
paper_authors.avatarUrl: /avatars/a54ddb6d69a73f63f9d848f28abcbff3.svg
paper_authors.fullname: PeihaoChen
paper_authors.type: user
paper_authors.user: PeihaoChen
paper_authors._id: 674e7370881f85b677414840
paper_authors.name: Junyan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:02:54.507Z
paper_authors._id: 62d09eb86a61a88ea0d83918
paper_authors.avatarUrl: /avatars/81b511d94cced304ffca058caff662d4.svg
paper_authors.fullname: Junyan Li
paper_authors.type: user
paper_authors.user: senfu
paper_authors._id: 674e7370881f85b677414841
paper_authors.name: Shuailei Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:02:31.843Z
paper_authors._id: 6458958a4b7baff9a8489623
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6458958a4b7baff9a8489623/vPV3NtHULvwH0bwx_FJm6.png
paper_authors.fullname: Shuailei Ma
paper_authors.type: user
paper_authors.user: Xiaomabufei
paper_authors._id: 674e7370881f85b677414842
paper_authors.name: Xinyu Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:02:25.815Z
paper_authors._id: 670d0fe6c9f167bb20e81508
paper_authors.avatarUrl: /avatars/35b13b38a8ffdaf9ef06a9c6c8bd90ad.svg
paper_authors.fullname: Xinyu Sun
paper_authors.type: user
paper_authors.user: Sunxy111
paper_authors._id: 674e7370881f85b677414843
paper_authors.name: Tianhang Xiang
paper_authors._id: 674e7370881f85b677414844
paper_authors.name: Yinjie Lei
paper_authors._id: 674e7370881f85b677414845
paper_authors.name: Mingkui Tan
paper_authors._id: 674e7370881f85b677414846
paper_authors.name: Chuang Gan
paper_title: LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual
    Preferences
paper_summary: Research on 3D Vision-Language Models (3D-VLMs) is gaining increasing
  attention, which is crucial for developing embodied AI within 3D scenes, such
  as visual navigation and embodied question answering. Due to the high density
  of visual features, especially in large 3D scenes, accurately locating
  task-relevant visual information is challenging. Existing works attempt to
  segment all objects and consider their features as scene representations.
  However, these task-agnostic object features include much redundant information
  and missing details for the task-relevant area. To tackle these problems, we
  propose LSceneLLM, an adaptive framework that automatically identifies
  task-relevant areas by leveraging LLM's visual preference for different tasks,
  followed by a plug-and-play scene magnifier module to capture fine-grained
  details in focused areas. Specifically, a dense token selector examines the
  attention map of LLM to identify visual preferences for the instruction input.
  It then magnifies fine-grained details of the focusing area. An adaptive
  self-attention module is leveraged to fuse the coarse-grained and selected
  fine-grained visual information. To comprehensively evaluate the large scene
  understanding ability of 3D-VLMs, we further introduce a cross-room
  understanding benchmark, XR-Scene, which contains a series of large scene
  understanding tasks including XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption.
  Experiments show that our method surpasses existing methods on both large scene
  understanding and existing scene understanding benchmarks. Plunging our scene
  magnifier module into the existing 3D-VLMs also brings significant improvement.
paper_ai_keywords: LSceneLLM dense token selector attention map plug-and-play scene magnifier module adaptive self-attention module XR-Scene XR-QA XR-EmbodiedPlanning XR-SceneCaption