Are Vision-Language Models Truly Understanding Multi-vision Sensor?
Large-scale Vision-Language Models (VLMs) have advanced by aligning vision
inputs with text, significantly improving performance in computer vision tasks.
Moreover, for VLMs to be effectively utilized in real-world applications, an
understanding of diverse multi-vision sensor data, such as thermal, depth, and
X-ray information, is essential. However, we find that current VLMs process
multi-vision sensor images without deep understanding of sensor information,
disregarding each sensor's unique physical properties. This limitation
restricts their capacity to interpret and respond to complex questions
requiring multi-vision sensor reasoning. To address this, we propose a novel
Multi-vision Sensor Perception and Reasoning (MS-PR) benchmark, assessing VLMs
on their capacity for sensor-specific reasoning. Moreover, we introduce Diverse
Negative Attributes (DNA) optimization to enable VLMs to perform deep reasoning
on multi-vision sensor tasks, helping to bridge the core information gap
between images and sensor data. Extensive experimental results validate that
the proposed DNA method can significantly improve the multi-vision sensor
reasoning for VLMs.
67742e073adba6b02dd84e3e
Sangyun Chung
claimed_verified
2025-01-03T14:05:03.795Z
65a4bf8e90b5e87bcdff41c7
https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
Sangyun Chung
user
topyun
67742e073adba6b02dd84e3f
Youngjoon Yu
67742e073adba6b02dd84e40
Youngchae Chee
67742e073adba6b02dd84e41
Se Yeon Kim
67742e073adba6b02dd84e42
Byung-Kwan Lee
claimed_verified
2025-06-10T09:31:25.638Z
657152eb12f162153b50ec9d
https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
Byung-Kwan Lee
user
BK-Lee
67742e073adba6b02dd84e43
Yong Man Ro
Are Vision-Language Models Truly Understanding Multi-vision Sensor?
Large-scale Vision-Language Models (VLMs) have advanced by aligning vision
inputs with text, significantly improving performance in computer vision tasks.
Moreover, for VLMs to be effectively utilized in real-world applications, an
understanding of diverse multi-vision sensor data, such as thermal, depth, and
X-ray information, is essential. However, we find that current VLMs process
multi-vision sensor images without deep understanding of sensor information,
disregarding each sensor's unique physical properties. This limitation
restricts their capacity to interpret and respond to complex questions
requiring multi-vision sensor reasoning. To address this, we propose a novel
Multi-vision Sensor Perception and Reasoning (MS-PR) benchmark, assessing VLMs
on their capacity for sensor-specific reasoning. Moreover, we introduce Diverse
Negative Attributes (DNA) optimization to enable VLMs to perform deep reasoning
on multi-vision sensor tasks, helping to bridge the core information gap
between images and sensor data. Extensive experimental results validate that
the proposed DNA method can significantly improve the multi-vision sensor
reasoning for VLMs.
A benchmark and optimization method are proposed to enhance(VLMs) ability to reason with multi-vision sensor data, improving their performance on tasks requiring sensor-specific understanding.
VLMs Multi-vision Sensor Perception and Reasoning (MS-PR) Diverse Negative Attributes (DNA) sensor-specific reasoning thermal depth X-ray information