LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual
  Preferences
Research on 3D Vision-Language Models (3D-VLMs) is gaining increasing
attention, which is crucial for developing embodied AI within 3D scenes, such
as visual navigation and embodied question answering. Due to the high density
of visual features, especially in large 3D scenes, accurately locating
task-relevant visual information is challenging. Existing works attempt to
segment all objects and consider their features as scene representations.
However, these task-agnostic object features include much redundant information
and missing details for the task-relevant area. To tackle these problems, we
propose LSceneLLM, an adaptive framework that automatically identifies
task-relevant areas by leveraging LLM's visual preference for different tasks,
followed by a plug-and-play scene magnifier module to capture fine-grained
details in focused areas. Specifically, a dense token selector examines the
attention map of LLM to identify visual preferences for the instruction input.
It then magnifies fine-grained details of the focusing area. An adaptive
self-attention module is leveraged to fuse the coarse-grained and selected
fine-grained visual information. To comprehensively evaluate the large scene
understanding ability of 3D-VLMs, we further introduce a cross-room
understanding benchmark, XR-Scene, which contains a series of large scene
understanding tasks including XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption.
Experiments show that our method surpasses existing methods on both large scene
understanding and existing scene understanding benchmarks. Plunging our scene
magnifier module into the existing 3D-VLMs also brings significant improvement.
674e7370881f85b67741483e
Hongyan Zhi
claimed_verified
2024-12-03T11:15:16.287Z
674b2406591d7232820252cd
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bhDlUVkFQ66yt3BEVs6WU.png
Hongyan Zhi
user
Hoyard
674e7370881f85b67741483f
Peihao Chen
admin_assigned
2024-12-04T12:03:06.868Z
64bfe895c874dda773303b44
/avatars/a54ddb6d69a73f63f9d848f28abcbff3.svg
PeihaoChen
user
PeihaoChen
674e7370881f85b677414840
Junyan Li
admin_assigned
2024-12-04T12:02:54.507Z
62d09eb86a61a88ea0d83918
/avatars/81b511d94cced304ffca058caff662d4.svg
Junyan Li
user
senfu
674e7370881f85b677414841
Shuailei Ma
admin_assigned
2024-12-04T12:02:31.843Z
6458958a4b7baff9a8489623
https://cdn-avatars.huggingface.co/v1/production/uploads/6458958a4b7baff9a8489623/vPV3NtHULvwH0bwx_FJm6.png
Shuailei Ma
user
Xiaomabufei
674e7370881f85b677414842
Xinyu Sun
admin_assigned
2024-12-04T12:02:25.815Z
670d0fe6c9f167bb20e81508
/avatars/35b13b38a8ffdaf9ef06a9c6c8bd90ad.svg
Xinyu Sun
user
Sunxy111
674e7370881f85b677414843
Tianhang Xiang
674e7370881f85b677414844
Yinjie Lei
674e7370881f85b677414845
Mingkui Tan
674e7370881f85b677414846
Chuang Gan
LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual
  Preferences
Research on 3D Vision-Language Models (3D-VLMs) is gaining increasing
attention, which is crucial for developing embodied AI within 3D scenes, such
as visual navigation and embodied question answering. Due to the high density
of visual features, especially in large 3D scenes, accurately locating
task-relevant visual information is challenging. Existing works attempt to
segment all objects and consider their features as scene representations.
However, these task-agnostic object features include much redundant information
and missing details for the task-relevant area. To tackle these problems, we
propose LSceneLLM, an adaptive framework that automatically identifies
task-relevant areas by leveraging LLM's visual preference for different tasks,
followed by a plug-and-play scene magnifier module to capture fine-grained
details in focused areas. Specifically, a dense token selector examines the
attention map of LLM to identify visual preferences for the instruction input.
It then magnifies fine-grained details of the focusing area. An adaptive
self-attention module is leveraged to fuse the coarse-grained and selected
fine-grained visual information. To comprehensively evaluate the large scene
understanding ability of 3D-VLMs, we further introduce a cross-room
understanding benchmark, XR-Scene, which contains a series of large scene
understanding tasks including XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption.
Experiments show that our method surpasses existing methods on both large scene
understanding and existing scene understanding benchmarks. Plunging our scene
magnifier module into the existing 3D-VLMs also brings significant improvement.
LSceneLLM, an adaptive 3D Vision-Language Model, enhances large scene understanding by identifying task-relevant areas and magnifying fine-grained details, outperforming existing methods on various benchmarks.
Vision-Language Models LSceneLLM visual preference dense token selector attention map adaptive self-attention module cross-room understanding benchmark XR-Scene XR-QA XR-EmbodiedPlanning XR-SceneCaption large scene understanding