Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
Vision-language models (VLMs) are achieving increasingly strong performance
on multimodal tasks. However, reasoning capabilities remain limited
particularly for smaller VLMs, while those of large-language models (LLMs) have
seen numerous improvements. We propose a technique to transfer capabilities
from LLMs to VLMs. On the recently introduced ChartQA, our method obtains
state-of-the-art performance when applied on the PaLI3-5B VLM by
chen2023pali3, while also enabling much better performance on PlotQA
and FigureQA.
  We first improve the chart representation by continuing the pre-training
stage using an improved version of the chart-to-table translation task by
liu2023deplot. We then propose constructing a 20x larger dataset than
the original training set. To improve general reasoning capabilities and
improve numerical operations, we synthesize reasoning traces using the table
representation of charts. Lastly, our model is fine-tuned using the multitask
loss introduced by hsieh2023distilling.
  Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B
without using an upstream OCR system, while keeping inference time constant
compared to the PaLI3-5B baseline. When rationales are further refined with a
simple program-of-thought prompt chen2023program, our model outperforms
the recently introduced Gemini Ultra and GPT-4V.
65fa55c8e483037d83fa56cb
Victor Carbune
admin_assigned
2024-03-21T12:55:55.218Z
656054efc40c3a6a0d9d7b3e
/avatars/6d93e933df1cd59b836f5f79b5e938bf.svg
Victor Carbune
user
vcarbune
65fa55c8e483037d83fa56cc
Hassan Mansoor
admin_assigned
2024-03-21T12:55:48.884Z
629098fd5463575364e7697a
/avatars/e45f3ee28bff3571f76caf847d3c36db.svg
Hassan Mansoor
user
HassanMansoor
65fa55c8e483037d83fa56cd
Fangyu Liu
admin_assigned
2024-03-21T12:56:01.262Z
5f881856ee5616341bc51e67
https://cdn-avatars.huggingface.co/v1/production/uploads/5f881856ee5616341bc51e67/9UCZCuhBTpJC9tGPyGmMb.jpeg
Fangyu Liu
user
fl399
65fa55c8e483037d83fa56ce
Rahul Aralikatte
admin_assigned
2024-03-21T12:56:08.802Z
60270a7c32856987162c641a
https://cdn-avatars.huggingface.co/v1/production/uploads/60270a7c32856987162c641a/yNO2n0MMOiTqRaHFD2bv1.jpeg
Rahul Aralikatte
user
rahular
65fa55c8e483037d83fa56cf
Gilles Baechler
admin_assigned
2024-03-21T12:56:15.562Z
633bf74e9a0fb78266be26cf
/avatars/360e37dff6841ee97779743e94e62213.svg
Gilles Baechler
user
g-les
65fa55c8e483037d83fa56d0
Jindong Chen
admin_assigned
2024-03-21T12:56:28.070Z
635242b8b3678a43742baec6
/avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
chenjindong
user
chenjindong
65fa55c8e483037d83fa56d1
Abhanshu Sharma
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
Vision-language models (VLMs) are achieving increasingly strong performance
on multimodal tasks. However, reasoning capabilities remain limited
particularly for smaller VLMs, while those of large-language models (LLMs) have
seen numerous improvements. We propose a technique to transfer capabilities
from LLMs to VLMs. On the recently introduced ChartQA, our method obtains
state-of-the-art performance when applied on the PaLI3-5B VLM by
chen2023pali3, while also enabling much better performance on PlotQA
and FigureQA.
  We first improve the chart representation by continuing the pre-training
stage using an improved version of the chart-to-table translation task by
liu2023deplot. We then propose constructing a 20x larger dataset than
the original training set. To improve general reasoning capabilities and
improve numerical operations, we synthesize reasoning traces using the table
representation of charts. Lastly, our model is fine-tuned using the multitask
loss introduced by hsieh2023distilling.
  Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B
without using an upstream OCR system, while keeping inference time constant
compared to the PaLI3-5B baseline. When rationales are further refined with a
simple program-of-thought prompt chen2023program, our model outperforms
the recently introduced Gemini Ultra and GPT-4V.
A method transfers reasoning capabilities from large-language models to vision-language models, achieving state-of-the-art performance on ChartQA and superior performance on PlotQA and FigureQA by enhancing chart representation and training with a larger, synthesized dataset and multitask loss.
vision-language models large-language models ChartQA PaLI3-5B plotQA FigureQA chart-to-table translation reasoning traces multitask loss ChartPaLI-5B PaLIX-55B OCR system program-of-thought prompt Gemini Ultra GPT-4V