MAVIS: Mathematical Visual Instruction Tuning
Multi-modal Large Language Models (MLLMs) have recently emerged as a
significant focus in academia and industry. Despite their proficiency in
general multi-modal scenarios, the mathematical problem-solving capabilities in
visual contexts remain insufficiently explored. We identify three key areas
within MLLMs that need to be improved: visual encoding of math diagrams,
diagram-language alignment, and mathematical reasoning skills. This draws forth
an urgent demand for large-scale, high-quality data and training pipelines in
visual mathematics. In this paper, we propose MAVIS, the first MAthematical
VISual instruction tuning paradigm for MLLMs, involving a series of
mathematical visual datasets and specialized MLLMs. Targeting the three issues,
MAVIS contains three progressive training stages from scratch. First, we curate
MAVIS-Caption, consisting of 558K diagram-caption pairs, to fine-tune a
math-specific vision encoder (CLIP-Math) through contrastive learning, tailored
for improved diagram visual encoding. Second, we utilize MAVIS-Caption to align
the CLIP-Math with a large language model (LLM) by a projection layer,
enhancing vision-language alignment in mathematical domains. Third, we
introduce MAVIS-Instruct, including 900K meticulously collected and annotated
visual math problems, which is adopted to finally instruct-tune the MLLM for
robust mathematical reasoning skills. In MAVIS-Instruct, we incorporate
complete chain-of-thought (CoT) rationales for each problem, and minimize
textual redundancy, thereby concentrating the model towards the visual
elements. Data and Models are released at https://github.com/ZrrSkywalker/MAVIS
669096c23a86663c1e4c0839
Renrui Zhang
admin_assigned
2024-07-12T08:23:59.791Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
669096c23a86663c1e4c083a
Xinyu Wei
669096c23a86663c1e4c083b
Dongzhi Jiang
admin_assigned
2024-07-12T08:28:34.946Z
6349214f8146350b3a4c5cdf
/avatars/cfd24caac9a87efb528d0f4c375932bc.svg
Dongzhi Jiang
user
CaraJ
669096c23a86663c1e4c083c
Yichi Zhang
669096c23a86663c1e4c083d
Ziyu Guo
admin_assigned
2024-07-12T08:29:40.489Z
647d9ab61a1fcad2fdbf2d3d
/avatars/48c8aeae8979d2c87df8bde922437d62.svg
Ziyu Guo
user
ZiyuG
669096c23a86663c1e4c083e
Chengzhuo Tong
669096c23a86663c1e4c083f
Jiaming Liu
669096c23a86663c1e4c0840
Aojun Zhou
claimed_verified
2024-07-12T09:19:01.750Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
669096c23a86663c1e4c0841
Bin Wei
669096c23a86663c1e4c0842
Shanghang Zhang
669096c23a86663c1e4c0843
Peng Gao
669096c23a86663c1e4c0844
Hongsheng Li
admin_assigned
2024-07-12T08:32:03.293Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
MAVIS: Mathematical Visual Instruction Tuning
Multi-modal Large Language Models (MLLMs) have recently emerged as a
significant focus in academia and industry. Despite their proficiency in
general multi-modal scenarios, the mathematical problem-solving capabilities in
visual contexts remain insufficiently explored. We identify three key areas
within MLLMs that need to be improved: visual encoding of math diagrams,
diagram-language alignment, and mathematical reasoning skills. This draws forth
an urgent demand for large-scale, high-quality data and training pipelines in
visual mathematics. In this paper, we propose MAVIS, the first MAthematical
VISual instruction tuning paradigm for MLLMs, involving a series of
mathematical visual datasets and specialized MLLMs. Targeting the three issues,
MAVIS contains three progressive training stages from scratch. First, we curate
MAVIS-Caption, consisting of 558K diagram-caption pairs, to fine-tune a
math-specific vision encoder (CLIP-Math) through contrastive learning, tailored
for improved diagram visual encoding. Second, we utilize MAVIS-Caption to align
the CLIP-Math with a large language model (LLM) by a projection layer,
enhancing vision-language alignment in mathematical domains. Third, we
introduce MAVIS-Instruct, including 900K meticulously collected and annotated
visual math problems, which is adopted to finally instruct-tune the MLLM for
robust mathematical reasoning skills. In MAVIS-Instruct, we incorporate
complete chain-of-thought (CoT) rationales for each problem, and minimize
textual redundancy, thereby concentrating the model towards the visual
elements. Data and Models are released at https://github.com/ZrrSkywalker/MAVIS
The paper introduces MAVIS, a comprehensive framework for improving the visual mathematics problem-solving capabilities of multi-modal large language models through tailored datasets and specialized training stages.
multi-modal large language models mathematical problem-solving visual encoding diagram-language alignment mathematical reasoning skills MAVIS MAVIS-Caption MAVIS-Instruct contrastive learning chain-of-thought rationales