title: ChartMuseum: Testing Visual Reasoning Capabilities of Large
    Vision-Language Models
summary: Chart understanding presents a unique challenge for large vision-language
  models (LVLMs), as it requires the integration of sophisticated textual and
  visual reasoning capabilities. However, current LVLMs exhibit a notable
  imbalance between these skills, falling short on visual reasoning that is
  difficult to perform in text. We conduct a case study using a synthetic dataset
  solvable only through visual reasoning and show that model performance degrades
  significantly with increasing visual complexity, while human performance
  remains robust. We then introduce ChartMuseum, a new Chart Question Answering
  (QA) benchmark containing 1,162 expert-annotated questions spanning multiple
  reasoning types, curated from real-world charts across 184 sources,
  specifically built to evaluate complex visual and textual reasoning. Unlike
  prior chart understanding benchmarks -- where frontier models perform similarly
  and near saturation -- our benchmark exposes a substantial gap between model
  and human performance, while effectively differentiating model capabilities:
  although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro
  attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct
  achieves only 38.5%. Moreover, on questions requiring primarily visual
  reasoning, all models experience a 35%-55% performance drop from
  text-reasoning-heavy question performance. Lastly, our qualitative error
  analysis reveals specific categories of visual reasoning that are challenging
  for current LVLMs.
paper_authors._id: 682bf33a6f59c839338ffdd0
paper_authors.name: Liyan Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:38.292Z
paper_authors._id: 62c70672e7d825deaae41e5e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c70672e7d825deaae41e5e/ICCpeBwmQ1NsgWcjG-MEZ.png
paper_authors.fullname: Liyan Tang
paper_authors.type: user
paper_authors.user: lytang
paper_authors._id: 682bf33a6f59c839338ffdd1
paper_authors.name: Grace Kim
paper_authors._id: 682bf33a6f59c839338ffdd2
paper_authors.name: Xinyu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T09:03:40.939Z
paper_authors._id: 65f356f3244d1b4066bcc62b
paper_authors.avatarUrl: /avatars/9eedf36a95e6d573bd3f8c0f0d16a9d6.svg
paper_authors.fullname: Xinyu (Lucy) Zhao
paper_authors.type: user
paper_authors.user: xyz-zy
paper_authors._id: 682bf33a6f59c839338ffdd3
paper_authors.name: Thom Lake
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:43:44.062Z
paper_authors._id: 64a87c60b76bfd863e715cab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a87c60b76bfd863e715cab/cpAUTOTEwhgP29aw6AOWA.jpeg
paper_authors.fullname: Thom Lake
paper_authors.type: user
paper_authors.user: thomlake
paper_authors._id: 682bf33a6f59c839338ffdd4
paper_authors.name: Wenxuan Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T19:29:36.814Z
paper_authors._id: 6502b65106c324fee1089ad5
paper_authors.avatarUrl: /avatars/9e054900bc1dd9a178a427ca9cf939da.svg
paper_authors.fullname: Wenxuan D
paper_authors.type: user
paper_authors.user: wenwenD
paper_authors._id: 682bf33a6f59c839338ffdd5
paper_authors.name: Fangcong Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:43:29.872Z
paper_authors._id: 64efa8748602335a044cd97f
paper_authors.avatarUrl: /avatars/0ab5df922cb0ce4abe7aed35e7b9100c.svg
paper_authors.fullname: Fangcong Yin
paper_authors.type: user
paper_authors.user: fcyin
paper_authors._id: 682bf33a6f59c839338ffdd6
paper_authors.name: Prasann Singhal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:43:24.263Z
paper_authors._id: 613cb3e1c7a43c281cd417a2
paper_authors.avatarUrl: /avatars/69123ba49c2aa1cd9f3cc5746f4839dc.svg
paper_authors.fullname: Prasann Singhal
paper_authors.type: user
paper_authors.user: PrasannSinghal
paper_authors._id: 682bf33a6f59c839338ffdd7
paper_authors.name: Manya Wadhwa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:43:18.026Z
paper_authors._id: 655ab2ccc11dee7f7e6db119
paper_authors.avatarUrl: /avatars/1c13e338cd4cc0b4eb681ed8f33abf19.svg
paper_authors.fullname: Manya Wadhwa
paper_authors.type: user
paper_authors.user: wadhma
paper_authors._id: 682bf33a6f59c839338ffdd8
paper_authors.name: Zeyu Leo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:42:57.568Z
paper_authors._id: 6607b0d29d2edd43f74dec98
paper_authors.avatarUrl: /avatars/437b5cbc555bf6906c3f07495a903ab4.svg
paper_authors.fullname: Zeyu Leo Liu
paper_authors.type: user
paper_authors.user: leo-liuzy
paper_authors._id: 682bf33a6f59c839338ffdd9
paper_authors.name: Zayne Sprague
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:42:49.622Z
paper_authors._id: 64e78a03e3953cd90bcad620
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e78a03e3953cd90bcad620/Rj_-xLJUsxdRmNhvRbssq.jpeg
paper_authors.fullname: Zayne Sprague
paper_authors.type: user
paper_authors.user: Zaynes
paper_authors._id: 682bf33a6f59c839338ffdda
paper_authors.name: Ramya Namuduri
paper_authors._id: 682bf33a6f59c839338ffddb
paper_authors.name: Bodun Hu
paper_authors._id: 682bf33a6f59c839338ffddc
paper_authors.name: Juan Diego Rodriguez
paper_authors._id: 682bf33a6f59c839338ffddd
paper_authors.name: Puyuan Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:42:13.748Z
paper_authors._id: 6480706f5409aa3e3bbaee16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qi2IrGGu7rgQVB_cfPNhh.png
paper_authors.fullname: Puyuan Peng
paper_authors.type: user
paper_authors.user: pyp1
paper_authors._id: 682bf33a6f59c839338ffdde
paper_authors.name: Greg Durrett
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:42:07.638Z
paper_authors._id: 65be9918b54ab5b37d1b67a7
paper_authors.avatarUrl: /avatars/9953707affb6881724c8efb2abf0c668.svg
paper_authors.fullname: Greg Durrett
paper_authors.type: user
paper_authors.user: gregdurrett
paper_title: ChartMuseum: Testing Visual Reasoning Capabilities of Large
    Vision-Language Models
paper_summary: Chart understanding presents a unique challenge for large vision-language
  models (LVLMs), as it requires the integration of sophisticated textual and
  visual reasoning capabilities. However, current LVLMs exhibit a notable
  imbalance between these skills, falling short on visual reasoning that is
  difficult to perform in text. We conduct a case study using a synthetic dataset
  solvable only through visual reasoning and show that model performance degrades
  significantly with increasing visual complexity, while human performance
  remains robust. We then introduce ChartMuseum, a new Chart Question Answering
  (QA) benchmark containing 1,162 expert-annotated questions spanning multiple
  reasoning types, curated from real-world charts across 184 sources,
  specifically built to evaluate complex visual and textual reasoning. Unlike
  prior chart understanding benchmarks -- where frontier models perform similarly
  and near saturation -- our benchmark exposes a substantial gap between model
  and human performance, while effectively differentiating model capabilities:
  although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro
  attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct
  achieves only 38.5%. Moreover, on questions requiring primarily visual
  reasoning, all models experience a 35%-55% performance drop from
  text-reasoning-heavy question performance. Lastly, our qualitative error
  analysis reveals specific categories of visual reasoning that are challenging
  for current LVLMs.
paper_ai_summary: A new benchmark, ChartMuseum, highlights the underperformance of large vision-language models in chart question answering, particularly for visually complex questions, compared to human accuracy.
paper_ai_keywords: vision-language models LVLMs visual reasoning textual reasoning synthetic dataset ChartMuseum Chart Question Answering QA benchmark reasoning types Gemini-2.5-Pro Qwen2.5-VL-72B-Instruct qualitative error analysis