title: ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question
    Answering
summary: Charts are ubiquitous, as people often use them to analyze data, answer
  questions, and discover critical insights. However, performing complex
  analytical tasks with charts requires significant perceptual and cognitive
  effort. Chart Question Answering (CQA) systems automate this process by
  enabling models to interpret and reason with visual representations of data.
  However, existing benchmarks like ChartQA lack real-world diversity and have
  recently shown performance saturation with modern large vision-language models
  (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark
  that includes 1,341 charts from 157 diverse sources, spanning various chart
  types, including infographics and dashboards, and featuring 1,948 questions in
  various types, such as multiple-choice, conversational, hypothetical, and
  unanswerable questions, to better reflect real-world challenges. Our
  evaluations with 21 models show a substantial performance drop for LVLMs on
  ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on
  ChartQAPro, underscoring the complexity of chart reasoning. We complement our
  findings with detailed error analyses and ablation studies, identifying key
  challenges and opportunities for advancing LVLMs in chart understanding and
  reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.
paper_authors._id: 6801a137a199bc0f78da6930
paper_authors.name: Ahmed Masry
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-18T09:39:16.613Z
paper_authors._id: 63efd75a5c2ceb16fc6e98fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63efd75a5c2ceb16fc6e98fc/qoA4LKuLTEr7hx90i90UK.jpeg
paper_authors.fullname: Ahmed Masry
paper_authors.type: user
paper_authors.user: ahmed-masry
paper_authors._id: 6801a137a199bc0f78da6931
paper_authors.name: Mohammed Saidul Islam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-18T09:39:14.462Z
paper_authors._id: 624eb4c9058568da72ac0964
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649325306950-624eb4c9058568da72ac0964.png
paper_authors.fullname: Saidul Islam
paper_authors.type: user
paper_authors.user: 38saidul
paper_authors._id: 6801a137a199bc0f78da6932
paper_authors.name: Mahir Ahmed
paper_authors._id: 6801a137a199bc0f78da6933
paper_authors.name: Aayush Bajaj
paper_authors._id: 6801a137a199bc0f78da6934
paper_authors.name: Firoz Kabir
paper_authors._id: 6801a137a199bc0f78da6935
paper_authors.name: Aaryaman Kartha
paper_authors._id: 6801a137a199bc0f78da6936
paper_authors.name: Md Tahmid Rahman Laskar
paper_authors._id: 6801a137a199bc0f78da6937
paper_authors.name: Mizanur Rahman
paper_authors._id: 6801a137a199bc0f78da6938
paper_authors.name: Shadikur Rahman
paper_authors._id: 6801a137a199bc0f78da6939
paper_authors.name: Mehrad Shahmohammadi
paper_authors._id: 6801a137a199bc0f78da693a
paper_authors.name: Megh Thakkar
paper_authors._id: 6801a137a199bc0f78da693b
paper_authors.name: Md Rizwan Parvez
paper_authors._id: 6801a137a199bc0f78da693c
paper_authors.name: Enamul Hoque
paper_authors._id: 6801a137a199bc0f78da693d
paper_authors.name: Shafiq Joty
paper_title: ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question
    Answering
paper_summary: Charts are ubiquitous, as people often use them to analyze data, answer
  questions, and discover critical insights. However, performing complex
  analytical tasks with charts requires significant perceptual and cognitive
  effort. Chart Question Answering (CQA) systems automate this process by
  enabling models to interpret and reason with visual representations of data.
  However, existing benchmarks like ChartQA lack real-world diversity and have
  recently shown performance saturation with modern large vision-language models
  (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark
  that includes 1,341 charts from 157 diverse sources, spanning various chart
  types, including infographics and dashboards, and featuring 1,948 questions in
  various types, such as multiple-choice, conversational, hypothetical, and
  unanswerable questions, to better reflect real-world challenges. Our
  evaluations with 21 models show a substantial performance drop for LVLMs on
  ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on
  ChartQAPro, underscoring the complexity of chart reasoning. We complement our
  findings with detailed error analyses and ablation studies, identifying key
  challenges and opportunities for advancing LVLMs in chart understanding and
  reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.
paper_ai_summary: ChartQAPro introduces a diverse benchmark for evaluating the performance of large vision-language models in chart reasoning, revealing significant performance gaps compared to existing benchmarks.
paper_ai_keywords: Chart Question Answering CQA large vision-language models LVLMs infographics dashboards multiple-choice conversational hypothetical unanswerable questions error analysis ablation studies chart reasoning