title: ColorBench: Can VLMs See and Understand the Colorful World? A
    Comprehensive Benchmark for Color Perception, Reasoning, and Robustness
summary: Color plays an important role in human perception and usually provides
  critical clues in visual reasoning. However, it is unclear whether and how
  vision-language models (VLMs) can perceive, understand, and leverage color as
  humans. This paper introduces ColorBench, an innovative benchmark meticulously
  crafted to assess the capabilities of VLMs in color understanding, including
  color perception, reasoning, and robustness. By curating a suite of diverse
  test scenarios, with grounding in real applications, ColorBench evaluates how
  these models perceive colors, infer meanings from color-based cues, and
  maintain consistent performance under varying color transformations. Through an
  extensive evaluation of 32 VLMs with varying language models and vision
  encoders, our paper reveals some undiscovered findings: (i) The scaling law
  (larger models are better) still holds on ColorBench, while the language model
  plays a more important role than the vision encoder. (ii) However, the
  performance gaps across models are relatively small, indicating that color
  understanding has been largely neglected by existing VLMs. (iii) CoT reasoning
  improves color understanding accuracies and robustness, though they are
  vision-centric tasks. (iv) Color clues are indeed leveraged by VLMs on
  ColorBench but they can also mislead models in some tasks. These findings
  highlight the critical limitations of current VLMs and underscore the need to
  enhance color comprehension. Our ColorBenchcan serve as a foundational tool for
  advancing the study of human-level color understanding of multimodal AI.
paper_authors._id: 67ffedb8b0c26d6ec0b608cf
paper_authors.name: Yijun Liang
paper_authors._id: 67ffedb8b0c26d6ec0b608d0
paper_authors.name: Ming Li
paper_authors._id: 67ffedb8b0c26d6ec0b608d1
paper_authors.name: Chenrui Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-17T08:05:05.662Z
paper_authors._id: 64a8121e35fab7cd04c30ed0
paper_authors.avatarUrl: /avatars/48849b84703158772f1022932331b143.svg
paper_authors.fullname: Chenrui Fan
paper_authors.type: user
paper_authors.user: Fcr09
paper_authors._id: 67ffedb8b0c26d6ec0b608d2
paper_authors.name: Ziyue Li
paper_authors._id: 67ffedb8b0c26d6ec0b608d3
paper_authors.name: Dang Nguyen
paper_authors._id: 67ffedb8b0c26d6ec0b608d4
paper_authors.name: Kwesi Cobbina
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T08:08:36.225Z
paper_authors._id: 63f546e0fcf95ecac2b0ee3e
paper_authors.avatarUrl: /avatars/02a401bcff91cc473d9946bbb771a985.svg
paper_authors.fullname: Kwesi Cobbina
paper_authors.type: user
paper_authors.user: kweCobi
paper_authors._id: 67ffedb8b0c26d6ec0b608d5
paper_authors.name: Shweta Bhardwaj
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T08:08:29.680Z
paper_authors._id: 639d4b8d860db464ae35c3ab
paper_authors.avatarUrl: /avatars/ec0fa3e91593a03fc9fb611e66b30553.svg
paper_authors.fullname: Shweta Bhardwaj
paper_authors.type: user
paper_authors.user: shweta12
paper_authors._id: 67ffedb8b0c26d6ec0b608d6
paper_authors.name: Jiuhai Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T08:08:22.786Z
paper_authors._id: 6393847e3e30234ae798b7be
paper_authors.avatarUrl: /avatars/daeb8c37dff4432d837a69b87c196521.svg
paper_authors.fullname: JiuhaiChen
paper_authors.type: user
paper_authors.user: jiuhai
paper_authors._id: 67ffedb8b0c26d6ec0b608d7
paper_authors.name: Fuxiao Liu
paper_authors._id: 67ffedb8b0c26d6ec0b608d8
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-17T08:05:07.396Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: ColorBench: Can VLMs See and Understand the Colorful World? A
    Comprehensive Benchmark for Color Perception, Reasoning, and Robustness
paper_summary: Color plays an important role in human perception and usually provides
  critical clues in visual reasoning. However, it is unclear whether and how
  vision-language models (VLMs) can perceive, understand, and leverage color as
  humans. This paper introduces ColorBench, an innovative benchmark meticulously
  crafted to assess the capabilities of VLMs in color understanding, including
  color perception, reasoning, and robustness. By curating a suite of diverse
  test scenarios, with grounding in real applications, ColorBench evaluates how
  these models perceive colors, infer meanings from color-based cues, and
  maintain consistent performance under varying color transformations. Through an
  extensive evaluation of 32 VLMs with varying language models and vision
  encoders, our paper reveals some undiscovered findings: (i) The scaling law
  (larger models are better) still holds on ColorBench, while the language model
  plays a more important role than the vision encoder. (ii) However, the
  performance gaps across models are relatively small, indicating that color
  understanding has been largely neglected by existing VLMs. (iii) CoT reasoning
  improves color understanding accuracies and robustness, though they are
  vision-centric tasks. (iv) Color clues are indeed leveraged by VLMs on
  ColorBench but they can also mislead models in some tasks. These findings
  highlight the critical limitations of current VLMs and underscore the need to
  enhance color comprehension. Our ColorBenchcan serve as a foundational tool for
  advancing the study of human-level color understanding of multimodal AI.
paper_ai_summary: ColorBench evaluates vision-language models' color perception, reasoning, and robustness, revealing limitations and emphasizing the need for enhanced color comprehension in multimodal AI.
paper_ai_keywords: vision-language models VLMs color understanding color perception color reasoning color robustness scaling law CoT reasoning