title: SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for
    Large-scale Vision-Language Models
summary: Large-scale Vision-Language Models (LVLMs) have significantly advanced with
  text-aligned vision inputs. They have made remarkable progress in computer
  vision tasks by aligning text modality with vision inputs. There are also
  endeavors to incorporate multi-vision sensors beyond RGB, including thermal,
  depth, and medical X-ray images. However, we observe that current LVLMs view
  images taken from multi-vision sensors as if they were in the same RGB domain
  without considering the physical characteristics of multi-vision sensors. They
  fail to convey the fundamental multi-vision sensor information from the dataset
  and the corresponding contextual knowledge properly. Consequently, alignment
  between the information from the actual physical environment and the text is
  not achieved correctly, making it difficult to answer complex sensor-related
  questions that consider the physical environment. In this paper, we aim to
  establish a multi-vision Sensor Perception And Reasoning benchmarK called SPARK
  that can reduce the fundamental multi-vision sensor information gap between
  images and multi-vision sensors. We generated 6,248 vision-language test
  samples automatically to investigate multi-vision sensory perception and
  multi-vision sensory reasoning on physical sensor knowledge proficiency across
  different formats, covering different types of sensor-related questions. We
  utilized these samples to assess ten leading LVLMs. The results showed that
  most models displayed deficiencies in multi-vision sensory reasoning to varying
  extents. Codes and data are available at https://github.com/top-yun/SPARK
paper_authors._id: 66c7f6b9d5a538bbb9e3b387
paper_authors.name: Youngjoon Yu
paper_authors._id: 66c7f6b9d5a538bbb9e3b388
paper_authors.name: Sangyun Chung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:23:41.928Z
paper_authors._id: 65a4bf8e90b5e87bcdff41c7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
paper_authors.fullname: Sangyun Chung
paper_authors.type: user
paper_authors.user: topyun
paper_authors._id: 66c7f6b9d5a538bbb9e3b389
paper_authors.name: Byung-Kwan Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-26T07:15:42.502Z
paper_authors._id: 657152eb12f162153b50ec9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
paper_authors.fullname: Byung-Kwan Lee
paper_authors.type: user
paper_authors.user: BK-Lee
paper_authors._id: 66c7f6b9d5a538bbb9e3b38a
paper_authors.name: Yong Man Ro
paper_title: SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for
    Large-scale Vision-Language Models
paper_summary: Large-scale Vision-Language Models (LVLMs) have significantly advanced with
  text-aligned vision inputs. They have made remarkable progress in computer
  vision tasks by aligning text modality with vision inputs. There are also
  endeavors to incorporate multi-vision sensors beyond RGB, including thermal,
  depth, and medical X-ray images. However, we observe that current LVLMs view
  images taken from multi-vision sensors as if they were in the same RGB domain
  without considering the physical characteristics of multi-vision sensors. They
  fail to convey the fundamental multi-vision sensor information from the dataset
  and the corresponding contextual knowledge properly. Consequently, alignment
  between the information from the actual physical environment and the text is
  not achieved correctly, making it difficult to answer complex sensor-related
  questions that consider the physical environment. In this paper, we aim to
  establish a multi-vision Sensor Perception And Reasoning benchmarK called SPARK
  that can reduce the fundamental multi-vision sensor information gap between
  images and multi-vision sensors. We generated 6,248 vision-language test
  samples automatically to investigate multi-vision sensory perception and
  multi-vision sensory reasoning on physical sensor knowledge proficiency across
  different formats, covering different types of sensor-related questions. We
  utilized these samples to assess ten leading LVLMs. The results showed that
  most models displayed deficiencies in multi-vision sensory reasoning to varying
  extents. Codes and data are available at https://github.com/top-yun/SPARK
paper_ai_keywords: Large-scale Vision-Language Models (LVLMs) text-aligned vision inputs RGB thermal depth medical X-ray images representation physical characteristics multi-vision sensor information contextual knowledge alignment physical environment multi-vision Sensor Perception And Reasoning benchmark SPARK vision-language test samples multi-vision sensory perception multi-vision sensory reasoning sensor-related questions leading LVLMs multi-vision sensory reasoning deficiencies physical sensor knowledge proficiency