3D Question Answering for City Scene Understanding
3D multimodal question answering (MQA) plays a crucial role in scene
understanding by enabling intelligent agents to comprehend their surroundings
in 3D environments. While existing research has primarily focused on indoor
household tasks and outdoor roadside autonomous driving tasks, there has been
limited exploration of city-level scene understanding tasks. Furthermore,
existing research faces challenges in understanding city scenes, due to the
absence of spatial semantic information and human-environment interaction
information at the city level.To address these challenges, we investigate 3D
MQA from both dataset and method perspectives. From the dataset perspective, we
introduce a novel 3D MQA dataset named City-3DQA for city-level scene
understanding, which is the first dataset to incorporate scene semantic and
human-environment interactive tasks within the city. From the method
perspective, we propose a Scene graph enhanced City-level Understanding method
(Sg-CityU), which utilizes the scene graph to introduce the spatial semantic. A
new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94
% and 63.76 % in different settings of City-3DQA. Compared to indoor 3D MQA
methods and zero-shot using advanced large language models (LLMs), Sg-CityU
demonstrates state-of-the-art (SOTA) performance in robustness and
generalization.
66a848766419b5b8fcea0163
Penglei Sun
admin_assigned
2024-07-30T09:01:33.517Z
64eded5fdfe0a679d840bc98
/avatars/4d4c67c13e547a4d296a301e8694e79e.svg
sunpenglei
user
sunpenglei
66a848766419b5b8fcea0164
Yaoxian Song
66a848766419b5b8fcea0165
Xiang Liu
claimed_verified
2024-07-30T07:25:52.460Z
63024676056ec3a2a8714b24
https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
Xiang Liu
user
Dominic789654
66a848766419b5b8fcea0166
Xiaofei Yang
66a848766419b5b8fcea0167
Qiang Wang
66a848766419b5b8fcea0168
Tiefeng Li
66a848766419b5b8fcea0169
Yang Yang
claimed_verified
2024-07-30T13:46:20.318Z
66a8e0fcb82318f0a13deafc
/avatars/a6794b5299a5af896b8d9843b20cef9d.svg
Yang Yang
user
yyiot
66a848766419b5b8fcea016a
Xiaowen Chu
admin_assigned
2024-07-30T09:03:05.982Z
6676935fcd0b89a0115174b0
/avatars/4caca1b672d29e787814f9a30bf20bcc.svg
Xiaowen Chu
user
wenxinsiju
3D Question Answering for City Scene Understanding
3D multimodal question answering (MQA) plays a crucial role in scene
understanding by enabling intelligent agents to comprehend their surroundings
in 3D environments. While existing research has primarily focused on indoor
household tasks and outdoor roadside autonomous driving tasks, there has been
limited exploration of city-level scene understanding tasks. Furthermore,
existing research faces challenges in understanding city scenes, due to the
absence of spatial semantic information and human-environment interaction
information at the city level.To address these challenges, we investigate 3D
MQA from both dataset and method perspectives. From the dataset perspective, we
introduce a novel 3D MQA dataset named City-3DQA for city-level scene
understanding, which is the first dataset to incorporate scene semantic and
human-environment interactive tasks within the city. From the method
perspective, we propose a Scene graph enhanced City-level Understanding method
(Sg-CityU), which utilizes the scene graph to introduce the spatial semantic. A
new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94
% and 63.76 % in different settings of City-3DQA. Compared to indoor 3D MQA
methods and zero-shot using advanced large language models (LLMs), Sg-CityU
demonstrates state-of-the-art (SOTA) performance in robustness and
generalization.
The research introduces a 3D multimodal question answering dataset and method for city-level scene understanding, incorporating spatial semantics and human-environment interactions, achieving state-of-the-art performance.
3D multimodal question answering 3D MQA city-level scene understanding scene graph City-3DQA Scene graph enhanced City-level Understanding Sg-CityU