title: ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
summary: Given the ubiquity of charts as a data analysis, visualization, and
  decision-making tool across industries and sciences, there has been a growing
  interest in developing pre-trained foundation models as well as general purpose
  instruction-tuned models for chart understanding and reasoning. However,
  existing methods suffer crucial drawbacks across two critical axes affecting
  the performance of chart representation models: they are trained on data
  generated from underlying data tables of the charts, ignoring the visual trends
  and patterns in chart images, and use weakly aligned vision-language backbone
  models for domain-specific training, limiting their generalizability when
  encountering charts in the wild. We address these important drawbacks and
  introduce ChartGemma, a novel chart understanding and reasoning model developed
  over PaliGemma. Rather than relying on underlying data tables, ChartGemma is
  trained on instruction-tuning data generated directly from chart images, thus
  capturing both high-level trends and low-level visual information from a
  diverse set of charts. Our simple approach achieves state-of-the-art results
  across 5 benchmarks spanning chart summarization, question answering, and
  fact-checking, and our elaborate qualitative studies on real-world charts show
  that ChartGemma generates more realistic and factually correct summaries
  compared to its contemporaries. We release the code, model checkpoints,
  dataset, and demos at https://github.com/vis-nlp/ChartGemma.
paper_authors._id: 668b4dc02c83140a9b2d539c
paper_authors.name: Ahmed Masry
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T20:00:52.109Z
paper_authors._id: 63efd75a5c2ceb16fc6e98fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63efd75a5c2ceb16fc6e98fc/qoA4LKuLTEr7hx90i90UK.jpeg
paper_authors.fullname: Ahmed Masry
paper_authors.type: user
paper_authors.user: ahmed-masry
paper_authors._id: 668b4dc02c83140a9b2d539d
paper_authors.name: Megh Thakkar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:08:21.042Z
paper_authors._id: 6500eb5e85a884a964d4b6af
paper_authors.avatarUrl: /avatars/275b78abdc9488a52eaf36176ef8f484.svg
paper_authors.fullname: Megh Thakkar
paper_authors.type: user
paper_authors.user: megh121198
paper_authors._id: 668b4dc02c83140a9b2d539e
paper_authors.name: Aayush Bajaj
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T20:00:50.406Z
paper_authors._id: 64ac1edfae8c893fcf78ef61
paper_authors.avatarUrl: /avatars/419305a5d6768d64287888001716b6d5.svg
paper_authors.fullname: Aayush Bajaj
paper_authors.type: user
paper_authors.user: theaayushbajaj
paper_authors._id: 668b4dc02c83140a9b2d539f
paper_authors.name: Aaryaman Kartha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:09:37.802Z
paper_authors._id: 64b9aab5008feb6c1ad9f042
paper_authors.avatarUrl: /avatars/6d1d5ed77cacafe0fe36f2bb7af84b01.svg
paper_authors.fullname: Aaryaman Kartha
paper_authors.type: user
paper_authors.user: aarykary123
paper_authors._id: 668b4dc02c83140a9b2d53a0
paper_authors.name: Enamul Hoque
paper_authors._id: 668b4dc02c83140a9b2d53a1
paper_authors.name: Shafiq Joty
paper_title: ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
paper_summary: Given the ubiquity of charts as a data analysis, visualization, and
  decision-making tool across industries and sciences, there has been a growing
  interest in developing pre-trained foundation models as well as general purpose
  instruction-tuned models for chart understanding and reasoning. However,
  existing methods suffer crucial drawbacks across two critical axes affecting
  the performance of chart representation models: they are trained on data
  generated from underlying data tables of the charts, ignoring the visual trends
  and patterns in chart images, and use weakly aligned vision-language backbone
  models for domain-specific training, limiting their generalizability when
  encountering charts in the wild. We address these important drawbacks and
  introduce ChartGemma, a novel chart understanding and reasoning model developed
  over PaliGemma. Rather than relying on underlying data tables, ChartGemma is
  trained on instruction-tuning data generated directly from chart images, thus
  capturing both high-level trends and low-level visual information from a
  diverse set of charts. Our simple approach achieves state-of-the-art results
  across 5 benchmarks spanning chart summarization, question answering, and
  fact-checking, and our elaborate qualitative studies on real-world charts show
  that ChartGemma generates more realistic and factually correct summaries
  compared to its contemporaries. We release the code, model checkpoints,
  dataset, and demos at https://github.com/vis-nlp/ChartGemma.
paper_ai_summary: ChartGemma, a novel chart understanding model, outperforms existing methods by training directly on chart images and using a vision-language backbone, achieving state-of-the-art results in summarization, question answering, and fact-checking tasks.
paper_ai_keywords: PaliGemma vision-language backbone chart summarization question answering fact-checking