title: CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal
    LLMs
summary: Chart understanding plays a pivotal role when applying Multimodal Large
  Language Models (MLLMs) to real-world tasks such as analyzing scientific papers
  or financial reports. However, existing datasets often focus on oversimplified
  and homogeneous charts with template-based questions, leading to an
  over-optimistic measure of progress. We demonstrate that although open-source
  models can appear to outperform strong proprietary models on these benchmarks,
  a simple stress test with slightly different charts or questions can
  deteriorate performance by up to 34.5%. In this work, we propose CharXiv, a
  comprehensive evaluation suite involving 2,323 natural, challenging, and
  diverse charts from arXiv papers. CharXiv includes two types of questions: 1)
  descriptive questions about examining basic chart elements and 2) reasoning
  questions that require synthesizing information across complex visual elements
  in the chart. To ensure quality, all charts and questions are handpicked,
  curated, and verified by human experts. Our results reveal a substantial,
  previously underestimated gap between the reasoning skills of the strongest
  proprietary model (i.e., GPT-4o), which achieves 47.1% accuracy, and the
  strongest open-source model (i.e., InternVL Chat V1.5), which achieves 29.2%.
  All models lag far behind human performance of 80.5%, underscoring weaknesses
  in the chart understanding capabilities of existing MLLMs. We hope CharXiv
  facilitates future research on MLLM chart understanding by providing a more
  realistic and faithful measure of progress. Project page and leaderboard:
  https://charxiv.github.io/
paper_authors._id: 667cbc2b557fe99a9eff62d9
paper_authors.name: Zirui Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T07:31:28.683Z
paper_authors._id: 641a38fdfb5ffff5ac78ceb0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/SOAogoOZQdPJJptuZZP9R.png
paper_authors.fullname: Zirui Wang
paper_authors.type: user
paper_authors.user: zwcolin
paper_authors._id: 667cbc2b557fe99a9eff62da
paper_authors.name: Mengzhou Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T14:20:32.941Z
paper_authors._id: 6241126dedf27fda71c6b573
paper_authors.avatarUrl: /avatars/ac106139c0e5967776f39e2755ca0792.svg
paper_authors.fullname: Mengzhou Xia
paper_authors.type: user
paper_authors.user: mengzhouxia
paper_authors._id: 667cbc2b557fe99a9eff62db
paper_authors.name: Luxi He
paper_authors._id: 667cbc2b557fe99a9eff62dc
paper_authors.name: Howard Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T15:36:51.913Z
paper_authors._id: 60878324a5da133ac6c3869b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60878324a5da133ac6c3869b/7ZEBNelx26UgDvT6g5FUM.png
paper_authors.fullname: Howard Chen
paper_authors.type: user
paper_authors.user: howard50b
paper_authors._id: 667cbc2b557fe99a9eff62dd
paper_authors.name: Yitao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T14:21:51.169Z
paper_authors._id: 65e7c3a2706749f643c3386e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/cAZCcbJRizBCR7vCsWZUw.jpeg
paper_authors.fullname: Yitao Liu
paper_authors.type: user
paper_authors.user: yitaoliu917
paper_authors._id: 667cbc2b557fe99a9eff62de
paper_authors.name: Richard Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T15:58:07.901Z
paper_authors._id: 6369b5d256d1f9349813278b
paper_authors.avatarUrl: /avatars/120119800de3f4c1e9d388e04d6b768a.svg
paper_authors.fullname: Richard Zhu
paper_authors.type: user
paper_authors.user: rich123
paper_authors._id: 667cbc2b557fe99a9eff62df
paper_authors.name: Kaiqu Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T14:22:15.810Z
paper_authors._id: 650237b5d1b0b0db4f29ae8a
paper_authors.avatarUrl: /avatars/8f92cf8f3f1ddb45c2c58c4a59ce4633.svg
paper_authors.fullname: KAIQU LIANG
paper_authors.type: user
paper_authors.user: kaiquliang
paper_authors._id: 667cbc2b557fe99a9eff62e0
paper_authors.name: Xindi Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T14:22:22.566Z
paper_authors._id: 613940c0905b1938233881e3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/613940c0905b1938233881e3/Vb4kFWKEq6AILUP9KmCZA.png
paper_authors.fullname: Xindi Wu
paper_authors.type: user
paper_authors.user: xindiw
paper_authors._id: 667cbc2b557fe99a9eff62e1
paper_authors.name: Haotian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T14:22:43.317Z
paper_authors._id: 63898b61ec1f539adc0f4da2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
paper_authors.fullname: Haotian Liu
paper_authors.type: user
paper_authors.user: liuhaotian
paper_authors._id: 667cbc2b557fe99a9eff62e2
paper_authors.name: Sadhika Malladi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T14:23:01.673Z
paper_authors._id: 6617f95a0ea62dc58800740a
paper_authors.avatarUrl: /avatars/264b05ebe8a74ffed8c5914bdcc6d5d3.svg
paper_authors.fullname: Sadhika Malladi
paper_authors.type: user
paper_authors.user: sadhikamalladi
paper_authors._id: 667cbc2b557fe99a9eff62e3
paper_authors.name: Alexis Chevalier
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T14:23:07.186Z
paper_authors._id: 63127c55f568fb0098f7e077
paper_authors.avatarUrl: /avatars/d3e3bce2e956aeea9ced3ace6173a14c.svg
paper_authors.fullname: Alexis Chevalier
paper_authors.type: user
paper_authors.user: AlexChvl
paper_authors._id: 667cbc2b557fe99a9eff62e4
paper_authors.name: Sanjeev Arora
paper_authors._id: 667cbc2b557fe99a9eff62e5
paper_authors.name: Danqi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T14:23:21.405Z
paper_authors._id: 653a4ca8669b77a9f8034ec0
paper_authors.avatarUrl: /avatars/005ba9ead6b95b9ee8a31d050de77cee.svg
paper_authors.fullname: Danqi Chen
paper_authors.type: user
paper_authors.user: cdq10131
paper_title: CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal
    LLMs
paper_summary: Chart understanding plays a pivotal role when applying Multimodal Large
  Language Models (MLLMs) to real-world tasks such as analyzing scientific papers
  or financial reports. However, existing datasets often focus on oversimplified
  and homogeneous charts with template-based questions, leading to an
  over-optimistic measure of progress. We demonstrate that although open-source
  models can appear to outperform strong proprietary models on these benchmarks,
  a simple stress test with slightly different charts or questions can
  deteriorate performance by up to 34.5%. In this work, we propose CharXiv, a
  comprehensive evaluation suite involving 2,323 natural, challenging, and
  diverse charts from arXiv papers. CharXiv includes two types of questions: 1)
  descriptive questions about examining basic chart elements and 2) reasoning
  questions that require synthesizing information across complex visual elements
  in the chart. To ensure quality, all charts and questions are handpicked,
  curated, and verified by human experts. Our results reveal a substantial,
  previously underestimated gap between the reasoning skills of the strongest
  proprietary model (i.e., GPT-4o), which achieves 47.1% accuracy, and the
  strongest open-source model (i.e., InternVL Chat V1.5), which achieves 29.2%.
  All models lag far behind human performance of 80.5%, underscoring weaknesses
  in the chart understanding capabilities of existing MLLMs. We hope CharXiv
  facilitates future research on MLLM chart understanding by providing a more
  realistic and faithful measure of progress. Project page and leaderboard:
  https://charxiv.github.io/
paper_ai_keywords: Multimodal Large Language Models (MLLMs) Chart understanding Scientific papers Financial reports Oversimplified and homogeneous charts Template-based questions Combination-optimistic measure Open-source models Strong proprietary models Stress test CharXiv Comprehensive evaluation suite Challenging and diverse charts Descriptive questions Reasoning questions Complex visual elements Handpicked charts and questions Human experts GPT-4o InternVL Chat V1.5 Reasoning skills Accuracy Human performance