MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal
  Mathematical Reasoning
Natural language image-caption datasets, widely used for training Large
Multimodal Models, mainly focus on natural scenarios and overlook the intricate
details of mathematical figures that are critical for problem-solving,
hindering the advancement of current LMMs in multimodal mathematical reasoning.
To this end, we propose leveraging code as supervision for cross-modal
alignment, since code inherently encodes all information needed to generate
corresponding figures, establishing a precise connection between the two
modalities. Specifically, we co-develop our image-to-code model and dataset
with model-in-the-loop approach, resulting in an image-to-code model,
FigCodifier and ImgCode-8.6M dataset, the largest image-code dataset to date.
Furthermore, we utilize FigCodifier to synthesize novel mathematical figures
and then construct MM-MathInstruct-3M, a high-quality multimodal math
instruction fine-tuning dataset. Finally, we present MathCoder-VL, trained with
ImgCode-8.6M for cross-modal alignment and subsequently fine-tuned on
MM-MathInstruct-3M for multimodal math problem solving. Our model achieves a
new open-source SOTA across all six metrics. Notably, it surpasses GPT-4o and
Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista,
achieving improvements of 8.9% and 9.2%. The dataset and models will be
released at https://github.com/mathllm/MathCoder.
68269484ae8200ca5aff96a3
Ke Wang
68269484ae8200ca5aff96a4
Junting Pan
68269484ae8200ca5aff96a5
Linda Wei
68269484ae8200ca5aff96a6
Aojun Zhou
claimed_verified
2025-05-28T09:01:29.817Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
68269484ae8200ca5aff96a7
Weikang Shi
68269484ae8200ca5aff96a8
Zimu Lu
68269484ae8200ca5aff96a9
Han Xiao
claimed_verified
2025-05-28T09:01:31.865Z
666aa99cd1652853e4f9a8b9
/avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
HanXiao
user
HanXiao1999
68269484ae8200ca5aff96aa
Yunqiao Yang
68269484ae8200ca5aff96ab
Houxing Ren
68269484ae8200ca5aff96ac
Mingjie Zhan
68269484ae8200ca5aff96ad
Hongsheng Li
MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal
  Mathematical Reasoning
Natural language image-caption datasets, widely used for training Large
Multimodal Models, mainly focus on natural scenarios and overlook the intricate
details of mathematical figures that are critical for problem-solving,
hindering the advancement of current LMMs in multimodal mathematical reasoning.
To this end, we propose leveraging code as supervision for cross-modal
alignment, since code inherently encodes all information needed to generate
corresponding figures, establishing a precise connection between the two
modalities. Specifically, we co-develop our image-to-code model and dataset
with model-in-the-loop approach, resulting in an image-to-code model,
FigCodifier and ImgCode-8.6M dataset, the largest image-code dataset to date.
Furthermore, we utilize FigCodifier to synthesize novel mathematical figures
and then construct MM-MathInstruct-3M, a high-quality multimodal math
instruction fine-tuning dataset. Finally, we present MathCoder-VL, trained with
ImgCode-8.6M for cross-modal alignment and subsequently fine-tuned on
MM-MathInstruct-3M for multimodal math problem solving. Our model achieves a
new open-source SOTA across all six metrics. Notably, it surpasses GPT-4o and
Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista,
achieving improvements of 8.9% and 9.2%. The dataset and models will be
released at https://github.com/mathllm/MathCoder.
A new image-to-code model and dataset are developed to enhance multimodal mathematical reasoning, resulting in a model that outperforms current benchmarks in mathematical problem-solving.
image-to-code model cross-modal alignment FigCodifier ImgCode-8.6M multimodal math instruction MM-MathInstruct-3M MathCoder-VL multimodal math problem solving