CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal
  LLMs
Chart understanding plays a pivotal role when applying Multimodal Large
Language Models (MLLMs) to real-world tasks such as analyzing scientific papers
or financial reports. However, existing datasets often focus on oversimplified
and homogeneous charts with template-based questions, leading to an
over-optimistic measure of progress. We demonstrate that although open-source
models can appear to outperform strong proprietary models on these benchmarks,
a simple stress test with slightly different charts or questions can
deteriorate performance by up to 34.5%. In this work, we propose CharXiv, a
comprehensive evaluation suite involving 2,323 natural, challenging, and
diverse charts from arXiv papers. CharXiv includes two types of questions: 1)
descriptive questions about examining basic chart elements and 2) reasoning
questions that require synthesizing information across complex visual elements
in the chart. To ensure quality, all charts and questions are handpicked,
curated, and verified by human experts. Our results reveal a substantial,
previously underestimated gap between the reasoning skills of the strongest
proprietary model (i.e., GPT-4o), which achieves 47.1% accuracy, and the
strongest open-source model (i.e., InternVL Chat V1.5), which achieves 29.2%.
All models lag far behind human performance of 80.5%, underscoring weaknesses
in the chart understanding capabilities of existing MLLMs. We hope CharXiv
facilitates future research on MLLM chart understanding by providing a more
realistic and faithful measure of progress. Project page and leaderboard:
https://charxiv.github.io/
667cbc2b557fe99a9eff62d9
Zirui Wang
claimed_verified
2024-06-27T07:31:28.683Z
641a38fdfb5ffff5ac78ceb0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/SOAogoOZQdPJJptuZZP9R.png
Zirui Wang
user
zwcolin
667cbc2b557fe99a9eff62da
Mengzhou Xia
admin_assigned
2024-06-27T14:20:32.941Z
6241126dedf27fda71c6b573
/avatars/ac106139c0e5967776f39e2755ca0792.svg
Mengzhou Xia
user
mengzhouxia
667cbc2b557fe99a9eff62db
Luxi He
667cbc2b557fe99a9eff62dc
Howard Chen
claimed_verified
2024-06-27T15:36:51.913Z
60878324a5da133ac6c3869b
https://cdn-avatars.huggingface.co/v1/production/uploads/60878324a5da133ac6c3869b/7ZEBNelx26UgDvT6g5FUM.png
Howard Chen
user
howard50b
667cbc2b557fe99a9eff62dd
Yitao Liu
admin_assigned
2024-06-27T14:21:51.169Z
65e7c3a2706749f643c3386e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/cAZCcbJRizBCR7vCsWZUw.jpeg
Yitao Liu
user
yitaoliu917
667cbc2b557fe99a9eff62de
Richard Zhu
claimed_verified
2024-06-27T15:58:07.901Z
6369b5d256d1f9349813278b
/avatars/7ed478f75d53b80d7656394afdf257c7.svg
Richard Zhu
user
rich123
667cbc2b557fe99a9eff62df
Kaiqu Liang
admin_assigned
2024-06-27T14:22:15.810Z
650237b5d1b0b0db4f29ae8a
/avatars/8f92cf8f3f1ddb45c2c58c4a59ce4633.svg
KAIQU LIANG
user
kaiquliang
667cbc2b557fe99a9eff62e0
Xindi Wu
admin_assigned
2024-06-27T14:22:22.566Z
613940c0905b1938233881e3
https://cdn-avatars.huggingface.co/v1/production/uploads/613940c0905b1938233881e3/Vb4kFWKEq6AILUP9KmCZA.png
Xindi Wu
user
xindiw
667cbc2b557fe99a9eff62e1
Haotian Liu
admin_assigned
2024-06-27T14:22:43.317Z
63898b61ec1f539adc0f4da2
https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
Haotian Liu
user
liuhaotian
667cbc2b557fe99a9eff62e2
Sadhika Malladi
admin_assigned
2024-06-27T14:23:01.673Z
6617f95a0ea62dc58800740a
/avatars/264b05ebe8a74ffed8c5914bdcc6d5d3.svg
Sadhika Malladi
user
sadhikamalladi
667cbc2b557fe99a9eff62e3
Alexis Chevalier
admin_assigned
2024-06-27T14:23:07.186Z
63127c55f568fb0098f7e077
/avatars/d3e3bce2e956aeea9ced3ace6173a14c.svg
Alexis Chevalier
user
AlexChvl
667cbc2b557fe99a9eff62e4
Sanjeev Arora
667cbc2b557fe99a9eff62e5
Danqi Chen
admin_assigned
2024-06-27T14:23:21.405Z
653a4ca8669b77a9f8034ec0
/avatars/005ba9ead6b95b9ee8a31d050de77cee.svg
Danqi Chen
user
cdq10131
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal
  LLMs
Chart understanding plays a pivotal role when applying Multimodal Large
Language Models (MLLMs) to real-world tasks such as analyzing scientific papers
or financial reports. However, existing datasets often focus on oversimplified
and homogeneous charts with template-based questions, leading to an
over-optimistic measure of progress. We demonstrate that although open-source
models can appear to outperform strong proprietary models on these benchmarks,
a simple stress test with slightly different charts or questions can
deteriorate performance by up to 34.5%. In this work, we propose CharXiv, a
comprehensive evaluation suite involving 2,323 natural, challenging, and
diverse charts from arXiv papers. CharXiv includes two types of questions: 1)
descriptive questions about examining basic chart elements and 2) reasoning
questions that require synthesizing information across complex visual elements
in the chart. To ensure quality, all charts and questions are handpicked,
curated, and verified by human experts. Our results reveal a substantial,
previously underestimated gap between the reasoning skills of the strongest
proprietary model (i.e., GPT-4o), which achieves 47.1% accuracy, and the
strongest open-source model (i.e., InternVL Chat V1.5), which achieves 29.2%.
All models lag far behind human performance of 80.5%, underscoring weaknesses
in the chart understanding capabilities of existing MLLMs. We hope CharXiv
facilitates future research on MLLM chart understanding by providing a more
realistic and faithful measure of progress. Project page and leaderboard:
https://charxiv.github.io/
CharXiv evaluates the chart understanding capabilities of MLLMs using a diverse and challenging dataset, uncovering significant weaknesses in model performance compared to human capabilities.
Multimodal Large Language Models MLLMs CharXiv descriptive questions reasoning questions chart understanding