title: Distill Visual Chart Reasoning Ability from LLMs to MLLMs
summary: Solving complex chart Q&A tasks requires advanced visual reasoning abilities
  in multimodal large language models (MLLMs). Recent studies highlight that
  these abilities consist of two main parts: recognizing key information from
  visual inputs and conducting reasoning over it. Thus, a promising approach to
  enhance MLLMs is to construct relevant training data focusing on the two
  aspects. However, collecting and annotating complex charts and questions is
  costly and time-consuming, and ensuring the quality of annotated answers
  remains a challenge. In this paper, we propose Code-as-Intermediary Translation
  (CIT), a cost-effective, efficient and easily scalable data synthesis method
  for distilling visual reasoning abilities from LLMs to MLLMs. The code serves
  as an intermediary that translates visual chart representations into textual
  representations, enabling LLMs to understand cross-modal information.
  Specifically, we employ text-based synthesizing techniques to construct
  chart-plotting code and produce ReachQA, a dataset containing 3k
  reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and
  reasoning abilities. Experiments show that when fine-tuned with our data,
  models not only perform well on chart-related benchmarks, but also demonstrate
  improved multimodal reasoning abilities on general mathematical benchmarks like
  MathVista. The code and dataset are publicly available at
  https://github.com/hewei2001/ReachQA.
paper_authors._id: 671b2777415094a3cb66e72c
paper_authors.name: Wei He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:28:48.057Z
paper_authors._id: 66ecee857264238429a1211f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66ecee857264238429a1211f/TbuM7ToLBrSxDF8mOccpK.jpeg
paper_authors.fullname: Wei He
paper_authors.type: user
paper_authors.user: hewei2001
paper_authors._id: 671b2777415094a3cb66e72d
paper_authors.name: Zhiheng Xi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:28:45.734Z
paper_authors._id: 653a6e5cae155b92bae77b74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
paper_authors.fullname: Zhiheng Xi
paper_authors.type: user
paper_authors.user: WooooDyy
paper_authors._id: 671b2777415094a3cb66e72e
paper_authors.name: Wanxu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:28:43.775Z
paper_authors._id: 64fece311499bba2b529254a
paper_authors.avatarUrl: /avatars/0d866448d90fc5a4921ed62b2d805260.svg
paper_authors.fullname: zwx
paper_authors.type: user
paper_authors.user: zh2001
paper_authors._id: 671b2777415094a3cb66e72f
paper_authors.name: Xiaoran Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-25T11:46:14.603Z
paper_authors._id: 6424fa5f988b8b19edba8ed4
paper_authors.avatarUrl: /avatars/cff4f1535632e7c8abc33fe7b862f60d.svg
paper_authors.fullname: Xiaoran Fan
paper_authors.type: user
paper_authors.user: cnxup
paper_authors._id: 671b2777415094a3cb66e730
paper_authors.name: Yiwen Ding
paper_authors._id: 671b2777415094a3cb66e731
paper_authors.name: Zifei Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-25T11:46:28.601Z
paper_authors._id: 66b1bbc8642b9e6d7a6e6db2
paper_authors.avatarUrl: /avatars/4916b8ceabad31d7dc55680f2f7523f0.svg
paper_authors.fullname: Zifei Shan
paper_authors.type: user
paper_authors.user: zifeishan
paper_authors._id: 671b2777415094a3cb66e732
paper_authors.name: Tao Gui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-25T11:46:35.070Z
paper_authors._id: 65b71b9e8bb894c96cc7707b
paper_authors.avatarUrl: /avatars/78e2a833054e89bcc34ddc6e789677d5.svg
paper_authors.fullname: Tao Gui
paper_authors.type: user
paper_authors.user: guixiaotao
paper_authors._id: 671b2777415094a3cb66e733
paper_authors.name: Qi Zhang
paper_authors._id: 671b2777415094a3cb66e734
paper_authors.name: Xuanjing Huang
paper_title: Distill Visual Chart Reasoning Ability from LLMs to MLLMs
paper_summary: Solving complex chart Q&A tasks requires advanced visual reasoning abilities
  in multimodal large language models (MLLMs). Recent studies highlight that
  these abilities consist of two main parts: recognizing key information from
  visual inputs and conducting reasoning over it. Thus, a promising approach to
  enhance MLLMs is to construct relevant training data focusing on the two
  aspects. However, collecting and annotating complex charts and questions is
  costly and time-consuming, and ensuring the quality of annotated answers
  remains a challenge. In this paper, we propose Code-as-Intermediary Translation
  (CIT), a cost-effective, efficient and easily scalable data synthesis method
  for distilling visual reasoning abilities from LLMs to MLLMs. The code serves
  as an intermediary that translates visual chart representations into textual
  representations, enabling LLMs to understand cross-modal information.
  Specifically, we employ text-based synthesizing techniques to construct
  chart-plotting code and produce ReachQA, a dataset containing 3k
  reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and
  reasoning abilities. Experiments show that when fine-tuned with our data,
  models not only perform well on chart-related benchmarks, but also demonstrate
  improved multimodal reasoning abilities on general mathematical benchmarks like
  MathVista. The code and dataset are publicly available at
  https://github.com/hewei2001/ReachQA.
paper_ai_summary: A method called Code-as-Intermediary Translation (CIT) uses code to translate visual charts into text, enhancing MLLMs with visual reasoning abilities through cost-effective data synthesis.
paper_ai_keywords: multimodal large language models (MLLMs) visual reasoning training data data synthesis chart-plotting code ReachQA dataset multimodal reasoning MathVista benchmarks