Distill Visual Chart Reasoning Ability from LLMs to MLLMs
Solving complex chart Q&A tasks requires advanced visual reasoning abilities
in multimodal large language models (MLLMs). Recent studies highlight that
these abilities consist of two main parts: recognizing key information from
visual inputs and conducting reasoning over it. Thus, a promising approach to
enhance MLLMs is to construct relevant training data focusing on the two
aspects. However, collecting and annotating complex charts and questions is
costly and time-consuming, and ensuring the quality of annotated answers
remains a challenge. In this paper, we propose Code-as-Intermediary Translation
(CIT), a cost-effective, efficient and easily scalable data synthesis method
for distilling visual reasoning abilities from LLMs to MLLMs. The code serves
as an intermediary that translates visual chart representations into textual
representations, enabling LLMs to understand cross-modal information.
Specifically, we employ text-based synthesizing techniques to construct
chart-plotting code and produce ReachQA, a dataset containing 3k
reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and
reasoning abilities. Experiments show that when fine-tuned with our data,
models not only perform well on chart-related benchmarks, but also demonstrate
improved multimodal reasoning abilities on general mathematical benchmarks like
MathVista. The code and dataset are publicly available at
https://github.com/hewei2001/ReachQA.
671b2777415094a3cb66e72c
Wei He
claimed_verified
2024-10-25T09:28:48.057Z
66ecee857264238429a1211f
https://cdn-avatars.huggingface.co/v1/production/uploads/66ecee857264238429a1211f/TbuM7ToLBrSxDF8mOccpK.jpeg
Wei He
user
hewei2001
671b2777415094a3cb66e72d
Zhiheng Xi
claimed_verified
2024-10-25T09:28:45.734Z
653a6e5cae155b92bae77b74
https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
Zhiheng Xi
user
WooooDyy
671b2777415094a3cb66e72e
Wanxu Zhao
claimed_verified
2024-10-25T09:28:43.775Z
64fece311499bba2b529254a
/avatars/0d866448d90fc5a4921ed62b2d805260.svg
zwx
user
zh2001
671b2777415094a3cb66e72f
Xiaoran Fan
admin_assigned
2024-10-25T11:46:14.603Z
6424fa5f988b8b19edba8ed4
/avatars/cff4f1535632e7c8abc33fe7b862f60d.svg
Xiaoran Fan
user
cnxup
671b2777415094a3cb66e730
Yiwen Ding
671b2777415094a3cb66e731
Zifei Shan
admin_assigned
2024-10-25T11:46:28.601Z
66b1bbc8642b9e6d7a6e6db2
/avatars/4916b8ceabad31d7dc55680f2f7523f0.svg
Zifei Shan
user
zifeishan
671b2777415094a3cb66e732
Tao Gui
admin_assigned
2024-10-25T11:46:35.070Z
65b71b9e8bb894c96cc7707b
/avatars/78e2a833054e89bcc34ddc6e789677d5.svg
Tao Gui
user
guixiaotao
671b2777415094a3cb66e733
Qi Zhang
671b2777415094a3cb66e734
Xuanjing Huang
Distill Visual Chart Reasoning Ability from LLMs to MLLMs
Solving complex chart Q&A tasks requires advanced visual reasoning abilities
in multimodal large language models (MLLMs). Recent studies highlight that
these abilities consist of two main parts: recognizing key information from
visual inputs and conducting reasoning over it. Thus, a promising approach to
enhance MLLMs is to construct relevant training data focusing on the two
aspects. However, collecting and annotating complex charts and questions is
costly and time-consuming, and ensuring the quality of annotated answers
remains a challenge. In this paper, we propose Code-as-Intermediary Translation
(CIT), a cost-effective, efficient and easily scalable data synthesis method
for distilling visual reasoning abilities from LLMs to MLLMs. The code serves
as an intermediary that translates visual chart representations into textual
representations, enabling LLMs to understand cross-modal information.
Specifically, we employ text-based synthesizing techniques to construct
chart-plotting code and produce ReachQA, a dataset containing 3k
reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and
reasoning abilities. Experiments show that when fine-tuned with our data,
models not only perform well on chart-related benchmarks, but also demonstrate
improved multimodal reasoning abilities on general mathematical benchmarks like
MathVista. The code and dataset are publicly available at
https://github.com/hewei2001/ReachQA.
A method called Code-as-Intermediary Translation (CIT) uses code to translate visual charts into text, enhancing MLLMs with visual reasoning abilities through cost-effective data synthesis.
multimodal large language models (MLLMs) visual reasoning training data data synthesis chart-plotting code ReachQA dataset multimodal reasoning MathVista benchmarks