title: Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
summary: Vision-language models (VLMs) are achieving increasingly strong performance
  on multimodal tasks. However, reasoning capabilities remain limited
  particularly for smaller VLMs, while those of large-language models (LLMs) have
  seen numerous improvements. We propose a technique to transfer capabilities
  from LLMs to VLMs. On the recently introduced ChartQA, our method obtains
  state-of-the-art performance when applied on the PaLI3-5B VLM by
  chen2023pali3, while also enabling much better performance on PlotQA
  and FigureQA.
    We first improve the chart representation by continuing the pre-training
  stage using an improved version of the chart-to-table translation task by
  liu2023deplot. We then propose constructing a 20x larger dataset than
  the original training set. To improve general reasoning capabilities and
  improve numerical operations, we synthesize reasoning traces using the table
  representation of charts. Lastly, our model is fine-tuned using the multitask
  loss introduced by hsieh2023distilling.
    Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B
  without using an upstream OCR system, while keeping inference time constant
  compared to the PaLI3-5B baseline. When rationales are further refined with a
  simple program-of-thought prompt chen2023program, our model outperforms
  the recently introduced Gemini Ultra and GPT-4V.
paper_authors._id: 65fa55c8e483037d83fa56cb
paper_authors.name: Victor Carbune
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:55:55.218Z
paper_authors._id: 656054efc40c3a6a0d9d7b3e
paper_authors.avatarUrl: /avatars/6d93e933df1cd59b836f5f79b5e938bf.svg
paper_authors.fullname: Victor Carbune
paper_authors.type: user
paper_authors.user: vcarbune
paper_authors._id: 65fa55c8e483037d83fa56cc
paper_authors.name: Hassan Mansoor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:55:48.884Z
paper_authors._id: 629098fd5463575364e7697a
paper_authors.avatarUrl: /avatars/e45f3ee28bff3571f76caf847d3c36db.svg
paper_authors.fullname: Hassan Mansoor
paper_authors.type: user
paper_authors.user: HassanMansoor
paper_authors._id: 65fa55c8e483037d83fa56cd
paper_authors.name: Fangyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:56:01.262Z
paper_authors._id: 5f881856ee5616341bc51e67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f881856ee5616341bc51e67/9UCZCuhBTpJC9tGPyGmMb.jpeg
paper_authors.fullname: Fangyu Liu
paper_authors.type: user
paper_authors.user: fl399
paper_authors._id: 65fa55c8e483037d83fa56ce
paper_authors.name: Rahul Aralikatte
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:56:08.802Z
paper_authors._id: 60270a7c32856987162c641a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60270a7c32856987162c641a/yNO2n0MMOiTqRaHFD2bv1.jpeg
paper_authors.fullname: Rahul Aralikatte
paper_authors.type: user
paper_authors.user: rahular
paper_authors._id: 65fa55c8e483037d83fa56cf
paper_authors.name: Gilles Baechler
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:56:15.562Z
paper_authors._id: 633bf74e9a0fb78266be26cf
paper_authors.avatarUrl: /avatars/360e37dff6841ee97779743e94e62213.svg
paper_authors.fullname: Gilles Baechler
paper_authors.type: user
paper_authors.user: g-les
paper_authors._id: 65fa55c8e483037d83fa56d0
paper_authors.name: Jindong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:56:28.070Z
paper_authors._id: 635242b8b3678a43742baec6
paper_authors.avatarUrl: /avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
paper_authors.fullname: chenjindong
paper_authors.type: user
paper_authors.user: chenjindong
paper_authors._id: 65fa55c8e483037d83fa56d1
paper_authors.name: Abhanshu Sharma
paper_title: Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
paper_summary: Vision-language models (VLMs) are achieving increasingly strong performance
  on multimodal tasks. However, reasoning capabilities remain limited
  particularly for smaller VLMs, while those of large-language models (LLMs) have
  seen numerous improvements. We propose a technique to transfer capabilities
  from LLMs to VLMs. On the recently introduced ChartQA, our method obtains
  state-of-the-art performance when applied on the PaLI3-5B VLM by
  chen2023pali3, while also enabling much better performance on PlotQA
  and FigureQA.
    We first improve the chart representation by continuing the pre-training
  stage using an improved version of the chart-to-table translation task by
  liu2023deplot. We then propose constructing a 20x larger dataset than
  the original training set. To improve general reasoning capabilities and
  improve numerical operations, we synthesize reasoning traces using the table
  representation of charts. Lastly, our model is fine-tuned using the multitask
  loss introduced by hsieh2023distilling.
    Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B
  without using an upstream OCR system, while keeping inference time constant
  compared to the PaLI3-5B baseline. When rationales are further refined with a
  simple program-of-thought prompt chen2023program, our model outperforms
  the recently introduced Gemini Ultra and GPT-4V.
paper_ai_summary: A method transfers reasoning capabilities from large-language models to vision-language models, achieving state-of-the-art performance on ChartQA and superior performance on PlotQA and FigureQA by enhancing chart representation and training with a larger, synthesized dataset and multitask loss.
paper_ai_keywords: vision-language models large-language models ChartQA PaLI3-5B plotQA FigureQA chart-to-table translation reasoning traces multitask loss ChartPaLI-5B PaLIX-55B OCR system program-of-thought prompt Gemini Ultra GPT-4V