title: Are Vision-Language Models Truly Understanding Multi-vision Sensor?
summary: Large-scale Vision-Language Models (VLMs) have advanced by aligning vision
  inputs with text, significantly improving performance in computer vision tasks.
  Moreover, for VLMs to be effectively utilized in real-world applications, an
  understanding of diverse multi-vision sensor data, such as thermal, depth, and
  X-ray information, is essential. However, we find that current VLMs process
  multi-vision sensor images without deep understanding of sensor information,
  disregarding each sensor's unique physical properties. This limitation
  restricts their capacity to interpret and respond to complex questions
  requiring multi-vision sensor reasoning. To address this, we propose a novel
  Multi-vision Sensor Perception and Reasoning (MS-PR) benchmark, assessing VLMs
  on their capacity for sensor-specific reasoning. Moreover, we introduce Diverse
  Negative Attributes (DNA) optimization to enable VLMs to perform deep reasoning
  on multi-vision sensor tasks, helping to bridge the core information gap
  between images and sensor data. Extensive experimental results validate that
  the proposed DNA method can significantly improve the multi-vision sensor
  reasoning for VLMs.
paper_authors._id: 67742e073adba6b02dd84e3e
paper_authors.name: Sangyun Chung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:03.795Z
paper_authors._id: 65a4bf8e90b5e87bcdff41c7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
paper_authors.fullname: Sangyun Chung
paper_authors.type: user
paper_authors.user: topyun
paper_authors._id: 67742e073adba6b02dd84e3f
paper_authors.name: Youngjoon Yu
paper_authors._id: 67742e073adba6b02dd84e40
paper_authors.name: Youngchae Chee
paper_authors._id: 67742e073adba6b02dd84e41
paper_authors.name: Se Yeon Kim
paper_authors._id: 67742e073adba6b02dd84e42
paper_authors.name: Byung-Kwan Lee
paper_authors._id: 67742e073adba6b02dd84e43
paper_authors.name: Yong Man Ro
paper_title: Are Vision-Language Models Truly Understanding Multi-vision Sensor?
paper_summary: Large-scale Vision-Language Models (VLMs) have advanced by aligning vision
  inputs with text, significantly improving performance in computer vision tasks.
  Moreover, for VLMs to be effectively utilized in real-world applications, an
  understanding of diverse multi-vision sensor data, such as thermal, depth, and
  X-ray information, is essential. However, we find that current VLMs process
  multi-vision sensor images without deep understanding of sensor information,
  disregarding each sensor's unique physical properties. This limitation
  restricts their capacity to interpret and respond to complex questions
  requiring multi-vision sensor reasoning. To address this, we propose a novel
  Multi-vision Sensor Perception and Reasoning (MS-PR) benchmark, assessing VLMs
  on their capacity for sensor-specific reasoning. Moreover, we introduce Diverse
  Negative Attributes (DNA) optimization to enable VLMs to perform deep reasoning
  on multi-vision sensor tasks, helping to bridge the core information gap
  between images and sensor data. Extensive experimental results validate that
  the proposed DNA method can significantly improve the multi-vision sensor
  reasoning for VLMs.
paper_ai_keywords: Large-scale Vision-Language Models (VLMs) Multi-vision Sensor Perception and Reasoning (MS-PR) Diverse Negative Attributes (DNA)