ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
Given the ubiquity of charts as a data analysis, visualization, and
decision-making tool across industries and sciences, there has been a growing
interest in developing pre-trained foundation models as well as general purpose
instruction-tuned models for chart understanding and reasoning. However,
existing methods suffer crucial drawbacks across two critical axes affecting
the performance of chart representation models: they are trained on data
generated from underlying data tables of the charts, ignoring the visual trends
and patterns in chart images, and use weakly aligned vision-language backbone
models for domain-specific training, limiting their generalizability when
encountering charts in the wild. We address these important drawbacks and
introduce ChartGemma, a novel chart understanding and reasoning model developed
over PaliGemma. Rather than relying on underlying data tables, ChartGemma is
trained on instruction-tuning data generated directly from chart images, thus
capturing both high-level trends and low-level visual information from a
diverse set of charts. Our simple approach achieves state-of-the-art results
across 5 benchmarks spanning chart summarization, question answering, and
fact-checking, and our elaborate qualitative studies on real-world charts show
that ChartGemma generates more realistic and factually correct summaries
compared to its contemporaries. We release the code, model checkpoints,
dataset, and demos at https://github.com/vis-nlp/ChartGemma.
668b4dc02c83140a9b2d539c
Ahmed Masry
claimed_verified
2024-07-08T20:00:52.109Z
63efd75a5c2ceb16fc6e98fc
https://cdn-avatars.huggingface.co/v1/production/uploads/63efd75a5c2ceb16fc6e98fc/qoA4LKuLTEr7hx90i90UK.jpeg
Ahmed Masry
user
ahmed-masry
668b4dc02c83140a9b2d539d
Megh Thakkar
claimed_verified
2024-07-18T09:08:21.042Z
6500eb5e85a884a964d4b6af
/avatars/275b78abdc9488a52eaf36176ef8f484.svg
Megh Thakkar
user
megh121198
668b4dc02c83140a9b2d539e
Aayush Bajaj
claimed_verified
2024-07-08T20:00:50.406Z
64ac1edfae8c893fcf78ef61
/avatars/419305a5d6768d64287888001716b6d5.svg
Aayush Bajaj
user
theaayushbajaj
668b4dc02c83140a9b2d539f
Aaryaman Kartha
admin_assigned
2024-07-08T21:09:37.802Z
64b9aab5008feb6c1ad9f042
/avatars/6d1d5ed77cacafe0fe36f2bb7af84b01.svg
Aaryaman Kartha
user
aarykary123
668b4dc02c83140a9b2d53a0
Enamul Hoque
668b4dc02c83140a9b2d53a1
Shafiq Joty
ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
Given the ubiquity of charts as a data analysis, visualization, and
decision-making tool across industries and sciences, there has been a growing
interest in developing pre-trained foundation models as well as general purpose
instruction-tuned models for chart understanding and reasoning. However,
existing methods suffer crucial drawbacks across two critical axes affecting
the performance of chart representation models: they are trained on data
generated from underlying data tables of the charts, ignoring the visual trends
and patterns in chart images, and use weakly aligned vision-language backbone
models for domain-specific training, limiting their generalizability when
encountering charts in the wild. We address these important drawbacks and
introduce ChartGemma, a novel chart understanding and reasoning model developed
over PaliGemma. Rather than relying on underlying data tables, ChartGemma is
trained on instruction-tuning data generated directly from chart images, thus
capturing both high-level trends and low-level visual information from a
diverse set of charts. Our simple approach achieves state-of-the-art results
across 5 benchmarks spanning chart summarization, question answering, and
fact-checking, and our elaborate qualitative studies on real-world charts show
that ChartGemma generates more realistic and factually correct summaries
compared to its contemporaries. We release the code, model checkpoints,
dataset, and demos at https://github.com/vis-nlp/ChartGemma.
ChartGemma, a novel chart understanding model, outperforms existing methods by training directly on chart images and using a vision-language backbone, achieving state-of-the-art results in summarization, question answering, and fact-checking tasks.
PaliGemma vision-language backbone chart summarization question answering fact-checking