ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
<|reference_start|>ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild: Given the ubiquity of charts as a data analysis, visualization, and decision-making tool across industries and sciences, there has been a growing interest in developing pre-trained foundation models as well as general purpose instruction-tuned models for chart understanding and reasoning. However, existing methods suffer crucial drawbacks across two critical axes affecting the performance of chart representation models: they are trained on data generated from underlying data tables of the charts, ignoring the visual trends and patterns in chart images, and use weakly aligned vision-language backbone models for domain-specific training, limiting their generalizability when encountering charts in the wild. We address these important drawbacks and introduce ChartGemma, a novel chart understanding and reasoning model developed over PaliGemma. Rather than relying on underlying data tables, ChartGemma is trained on instruction-tuning data generated directly from chart images, thus capturing both high-level trends and low-level visual information from a diverse set of charts. Our simple approach achieves state-of-the-art results across $5$ benchmarks spanning chart summarization, question answering, and fact-checking, and our elaborate qualitative studies on real-world charts show that ChartGemma generates more realistic and factually correct summaries compared to its contemporaries. We release the code, model checkpoints, dataset, and demos at https://github.com/vis-nlp/ChartGemma.<|reference_end|>
@article{masry2024chartgemma:,
    title={ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild},
    author={Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul
  Hoque, Shafiq Joty},
    journal={arXiv preprint arXiv:2407.04172},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.04172},
    primaryClass={cs.AI cs.CL cs.CV}
}