title: MAVIS: Mathematical Visual Instruction Tuning
summary: Multi-modal Large Language Models (MLLMs) have recently emerged as a
  significant focus in academia and industry. Despite their proficiency in
  general multi-modal scenarios, the mathematical problem-solving capabilities in
  visual contexts remain insufficiently explored. We identify three key areas
  within MLLMs that need to be improved: visual encoding of math diagrams,
  diagram-language alignment, and mathematical reasoning skills. This draws forth
  an urgent demand for large-scale, high-quality data and training pipelines in
  visual mathematics. In this paper, we propose MAVIS, the first MAthematical
  VISual instruction tuning paradigm for MLLMs, involving a series of
  mathematical visual datasets and specialized MLLMs. Targeting the three issues,
  MAVIS contains three progressive training stages from scratch. First, we curate
  MAVIS-Caption, consisting of 558K diagram-caption pairs, to fine-tune a
  math-specific vision encoder (CLIP-Math) through contrastive learning, tailored
  for improved diagram visual encoding. Second, we utilize MAVIS-Caption to align
  the CLIP-Math with a large language model (LLM) by a projection layer,
  enhancing vision-language alignment in mathematical domains. Third, we
  introduce MAVIS-Instruct, including 900K meticulously collected and annotated
  visual math problems, which is adopted to finally instruct-tune the MLLM for
  robust mathematical reasoning skills. In MAVIS-Instruct, we incorporate
  complete chain-of-thought (CoT) rationales for each problem, and minimize
  textual redundancy, thereby concentrating the model towards the visual
  elements. Data and Models are released at https://github.com/ZrrSkywalker/MAVIS
paper_authors._id: 669096c23a86663c1e4c0839
paper_authors.name: Renrui Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T08:23:59.791Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 669096c23a86663c1e4c083a
paper_authors.name: Xinyu Wei
paper_authors._id: 669096c23a86663c1e4c083b
paper_authors.name: Dongzhi Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T08:28:34.946Z
paper_authors._id: 6349214f8146350b3a4c5cdf
paper_authors.avatarUrl: /avatars/cfd24caac9a87efb528d0f4c375932bc.svg
paper_authors.fullname: Dongzhi Jiang
paper_authors.type: user
paper_authors.user: CaraJ
paper_authors._id: 669096c23a86663c1e4c083c
paper_authors.name: Yichi Zhang
paper_authors._id: 669096c23a86663c1e4c083d
paper_authors.name: Ziyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T08:29:40.489Z
paper_authors._id: 647d9ab61a1fcad2fdbf2d3d
paper_authors.avatarUrl: /avatars/48c8aeae8979d2c87df8bde922437d62.svg
paper_authors.fullname: Ziyu Guo
paper_authors.type: user
paper_authors.user: ZiyuG
paper_authors._id: 669096c23a86663c1e4c083e
paper_authors.name: Chengzhuo Tong
paper_authors._id: 669096c23a86663c1e4c083f
paper_authors.name: Jiaming Liu
paper_authors._id: 669096c23a86663c1e4c0840
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T09:19:01.750Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 669096c23a86663c1e4c0841
paper_authors.name: Bin Wei
paper_authors._id: 669096c23a86663c1e4c0842
paper_authors.name: Shanghang Zhang
paper_authors._id: 669096c23a86663c1e4c0843
paper_authors.name: Peng Gao
paper_authors._id: 669096c23a86663c1e4c0844
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T08:32:03.293Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: MAVIS: Mathematical Visual Instruction Tuning
paper_summary: Multi-modal Large Language Models (MLLMs) have recently emerged as a
  significant focus in academia and industry. Despite their proficiency in
  general multi-modal scenarios, the mathematical problem-solving capabilities in
  visual contexts remain insufficiently explored. We identify three key areas
  within MLLMs that need to be improved: visual encoding of math diagrams,
  diagram-language alignment, and mathematical reasoning skills. This draws forth
  an urgent demand for large-scale, high-quality data and training pipelines in
  visual mathematics. In this paper, we propose MAVIS, the first MAthematical
  VISual instruction tuning paradigm for MLLMs, involving a series of
  mathematical visual datasets and specialized MLLMs. Targeting the three issues,
  MAVIS contains three progressive training stages from scratch. First, we curate
  MAVIS-Caption, consisting of 558K diagram-caption pairs, to fine-tune a
  math-specific vision encoder (CLIP-Math) through contrastive learning, tailored
  for improved diagram visual encoding. Second, we utilize MAVIS-Caption to align
  the CLIP-Math with a large language model (LLM) by a projection layer,
  enhancing vision-language alignment in mathematical domains. Third, we
  introduce MAVIS-Instruct, including 900K meticulously collected and annotated
  visual math problems, which is adopted to finally instruct-tune the MLLM for
  robust mathematical reasoning skills. In MAVIS-Instruct, we incorporate
  complete chain-of-thought (CoT) rationales for each problem, and minimize
  textual redundancy, thereby concentrating the model towards the visual
  elements. Data and Models are released at https://github.com/ZrrSkywalker/MAVIS
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) mathematical problem-solving capabilities visual encoding diagram-language alignment mathematical reasoning skills CLIP-Math contrastive learning vision encoder vision-language alignment MAVIS-Caption MAVIS-Instruct chain-of-thought (CoT) rationales