title: MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal
    Mathematical Reasoning
summary: Natural language image-caption datasets, widely used for training Large
  Multimodal Models, mainly focus on natural scenarios and overlook the intricate
  details of mathematical figures that are critical for problem-solving,
  hindering the advancement of current LMMs in multimodal mathematical reasoning.
  To this end, we propose leveraging code as supervision for cross-modal
  alignment, since code inherently encodes all information needed to generate
  corresponding figures, establishing a precise connection between the two
  modalities. Specifically, we co-develop our image-to-code model and dataset
  with model-in-the-loop approach, resulting in an image-to-code model,
  FigCodifier and ImgCode-8.6M dataset, the largest image-code dataset to date.
  Furthermore, we utilize FigCodifier to synthesize novel mathematical figures
  and then construct MM-MathInstruct-3M, a high-quality multimodal math
  instruction fine-tuning dataset. Finally, we present MathCoder-VL, trained with
  ImgCode-8.6M for cross-modal alignment and subsequently fine-tuned on
  MM-MathInstruct-3M for multimodal math problem solving. Our model achieves a
  new open-source SOTA across all six metrics. Notably, it surpasses GPT-4o and
  Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista,
  achieving improvements of 8.9% and 9.2%. The dataset and models will be
  released at https://github.com/mathllm/MathCoder.
paper_authors._id: 68269484ae8200ca5aff96a3
paper_authors.name: Ke Wang
paper_authors._id: 68269484ae8200ca5aff96a4
paper_authors.name: Junting Pan
paper_authors._id: 68269484ae8200ca5aff96a5
paper_authors.name: Linda Wei
paper_authors._id: 68269484ae8200ca5aff96a6
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:29.817Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 68269484ae8200ca5aff96a7
paper_authors.name: Weikang Shi
paper_authors._id: 68269484ae8200ca5aff96a8
paper_authors.name: Zimu Lu
paper_authors._id: 68269484ae8200ca5aff96a9
paper_authors.name: Han Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:31.865Z
paper_authors._id: 666aa99cd1652853e4f9a8b9
paper_authors.avatarUrl: /avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
paper_authors.fullname: HanXiao
paper_authors.type: user
paper_authors.user: HanXiao1999
paper_authors._id: 68269484ae8200ca5aff96aa
paper_authors.name: Yunqiao Yang
paper_authors._id: 68269484ae8200ca5aff96ab
paper_authors.name: Houxing Ren
paper_authors._id: 68269484ae8200ca5aff96ac
paper_authors.name: Mingjie Zhan
paper_authors._id: 68269484ae8200ca5aff96ad
paper_authors.name: Hongsheng Li
paper_title: MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal
    Mathematical Reasoning
paper_summary: Natural language image-caption datasets, widely used for training Large
  Multimodal Models, mainly focus on natural scenarios and overlook the intricate
  details of mathematical figures that are critical for problem-solving,
  hindering the advancement of current LMMs in multimodal mathematical reasoning.
  To this end, we propose leveraging code as supervision for cross-modal
  alignment, since code inherently encodes all information needed to generate
  corresponding figures, establishing a precise connection between the two
  modalities. Specifically, we co-develop our image-to-code model and dataset
  with model-in-the-loop approach, resulting in an image-to-code model,
  FigCodifier and ImgCode-8.6M dataset, the largest image-code dataset to date.
  Furthermore, we utilize FigCodifier to synthesize novel mathematical figures
  and then construct MM-MathInstruct-3M, a high-quality multimodal math
  instruction fine-tuning dataset. Finally, we present MathCoder-VL, trained with
  ImgCode-8.6M for cross-modal alignment and subsequently fine-tuned on
  MM-MathInstruct-3M for multimodal math problem solving. Our model achieves a
  new open-source SOTA across all six metrics. Notably, it surpasses GPT-4o and
  Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista,
  achieving improvements of 8.9% and 9.2%. The dataset and models will be
  released at https://github.com/mathllm/MathCoder.
paper_ai_summary: A new image-to-code model and dataset are developed to enhance multimodal mathematical reasoning, resulting in a model that outperforms current benchmarks in mathematical problem-solving.
paper_ai_keywords: image-to-code model cross-modal alignment FigCodifier ImgCode-8.6M multimodal math instruction MM-MathInstruct-3M MathCoder-VL multimodal math problem solving