ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question
  Answering
Charts are ubiquitous, as people often use them to analyze data, answer
questions, and discover critical insights. However, performing complex
analytical tasks with charts requires significant perceptual and cognitive
effort. Chart Question Answering (CQA) systems automate this process by
enabling models to interpret and reason with visual representations of data.
However, existing benchmarks like ChartQA lack real-world diversity and have
recently shown performance saturation with modern large vision-language models
(LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark
that includes 1,341 charts from 157 diverse sources, spanning various chart
types, including infographics and dashboards, and featuring 1,948 questions in
various types, such as multiple-choice, conversational, hypothetical, and
unanswerable questions, to better reflect real-world challenges. Our
evaluations with 21 models show a substantial performance drop for LVLMs on
ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on
ChartQAPro, underscoring the complexity of chart reasoning. We complement our
findings with detailed error analyses and ablation studies, identifying key
challenges and opportunities for advancing LVLMs in chart understanding and
reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.
6801a137a199bc0f78da6930
Ahmed Masry
claimed_verified
2025-04-18T09:39:16.613Z
63efd75a5c2ceb16fc6e98fc
https://cdn-avatars.huggingface.co/v1/production/uploads/63efd75a5c2ceb16fc6e98fc/qoA4LKuLTEr7hx90i90UK.jpeg
Ahmed Masry
user
ahmed-masry
6801a137a199bc0f78da6931
Mohammed Saidul Islam
claimed_verified
2025-04-18T09:39:14.462Z
624eb4c9058568da72ac0964
https://cdn-avatars.huggingface.co/v1/production/uploads/1649325306950-624eb4c9058568da72ac0964.png
Saidul Islam
user
38saidul
6801a137a199bc0f78da6932
Mahir Ahmed
6801a137a199bc0f78da6933
Aayush Bajaj
6801a137a199bc0f78da6934
Firoz Kabir
6801a137a199bc0f78da6935
Aaryaman Kartha
6801a137a199bc0f78da6936
Md Tahmid Rahman Laskar
6801a137a199bc0f78da6937
Mizanur Rahman
6801a137a199bc0f78da6938
Shadikur Rahman
6801a137a199bc0f78da6939
Mehrad Shahmohammadi
6801a137a199bc0f78da693a
Megh Thakkar
6801a137a199bc0f78da693b
Md Rizwan Parvez
6801a137a199bc0f78da693c
Enamul Hoque
6801a137a199bc0f78da693d
Shafiq Joty
ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question
  Answering
Charts are ubiquitous, as people often use them to analyze data, answer
questions, and discover critical insights. However, performing complex
analytical tasks with charts requires significant perceptual and cognitive
effort. Chart Question Answering (CQA) systems automate this process by
enabling models to interpret and reason with visual representations of data.
However, existing benchmarks like ChartQA lack real-world diversity and have
recently shown performance saturation with modern large vision-language models
(LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark
that includes 1,341 charts from 157 diverse sources, spanning various chart
types, including infographics and dashboards, and featuring 1,948 questions in
various types, such as multiple-choice, conversational, hypothetical, and
unanswerable questions, to better reflect real-world challenges. Our
evaluations with 21 models show a substantial performance drop for LVLMs on
ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on
ChartQAPro, underscoring the complexity of chart reasoning. We complement our
findings with detailed error analyses and ablation studies, identifying key
challenges and opportunities for advancing LVLMs in chart understanding and
reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.
ChartQAPro introduces a diverse benchmark for evaluating the performance of large vision-language models in chart reasoning, revealing significant performance gaps compared to existing benchmarks.
Chart Question Answering CQA large vision-language models LVLMs infographics dashboards multiple-choice conversational hypothetical unanswerable questions error analysis ablation studies chart reasoning