title: InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced
    Mathematical Reasoning
summary: Pre-training on large-scale, high-quality datasets is crucial for enhancing
  the reasoning capabilities of Large Language Models (LLMs), especially in
  specialized domains such as mathematics. Despite the recognized importance, the
  Multimodal LLMs (MLLMs) field currently lacks a comprehensive open-source
  pre-training dataset specifically designed for mathematical reasoning. To
  address this gap, we introduce InfiMM-WebMath-40B, a high-quality dataset of
  interleaved image-text documents. It comprises 24 million web pages, 85 million
  associated image URLs, and 40 billion text tokens, all meticulously extracted
  and filtered from CommonCrawl. We provide a detailed overview of our data
  collection and processing pipeline. To demonstrate the robustness of
  InfiMM-WebMath-40B, we conducted evaluations in both text-only and multimodal
  settings. Our evaluations on text-only benchmarks show that, despite utilizing
  only 40 billion tokens, our dataset significantly enhances the performance of
  our 1.3B model, delivering results comparable to DeepSeekMath-1.3B, which uses
  120 billion tokens for the same model size. Nevertheless, with the introduction
  of our multi-modal math pre-training dataset, our models set a new
  state-of-the-art among open-source models on multi-modal math benchmarks such
  as MathVerse and We-Math. We release our data at
  https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.
paper_authors._id: 66ecd8f89cbe3098580d1a33
paper_authors.name: Xiaotian Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:46:01.294Z
paper_authors._id: 650dde4ce14eeb01d42b37a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png
paper_authors.fullname: Xiaotian Han
paper_authors.type: user
paper_authors.user: xiaotianhan
paper_authors._id: 66ecd8f89cbe3098580d1a34
paper_authors.name: Yiren Jian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T07:51:24.540Z
paper_authors._id: 64d2b7a76975d68147652a88
paper_authors.avatarUrl: /avatars/13be8f052c3f8067be70c1b64b0774f8.svg
paper_authors.fullname: Yiren Jian
paper_authors.type: user
paper_authors.user: yjian-bytedance
paper_authors._id: 66ecd8f89cbe3098580d1a35
paper_authors.name: Xuefeng Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T07:52:20.890Z
paper_authors._id: 65cb212725a3b8e44dc0a09a
paper_authors.avatarUrl: /avatars/66c8457a8a8a85e676654479f6860a4b.svg
paper_authors.fullname: Xuefeng Hu
paper_authors.type: user
paper_authors.user: bytehxf
paper_authors._id: 66ecd8f89cbe3098580d1a36
paper_authors.name: Haogeng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:45:59.371Z
paper_authors._id: 64f58c8bab97f9a83eca4963
paper_authors.avatarUrl: /avatars/b525fa19dcb6c051fe3ae260d85926e5.svg
paper_authors.fullname: haogengliu
paper_authors.type: user
paper_authors.user: lllliuhhhhggg
paper_authors._id: 66ecd8f89cbe3098580d1a37
paper_authors.name: Yiqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:45:57.179Z
paper_authors._id: 6566d9088556065b5268c666
paper_authors.avatarUrl: /avatars/85e275c3a1e70c21045f13b0cefc93fb.svg
paper_authors.fullname: Yiqi Wang
paper_authors.type: user
paper_authors.user: Yi-Qi638
paper_authors._id: 66ecd8f89cbe3098580d1a38
paper_authors.name: Qihang Fan
paper_authors._id: 66ecd8f89cbe3098580d1a39
paper_authors.name: Yuang Ai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:55:54.668Z
paper_authors._id: 65709464b3501cbcb8f4007e
paper_authors.avatarUrl: /avatars/4fb7524cb21e38138b4c67ff5501b403.svg
paper_authors.fullname: Yuang Ai
paper_authors.type: user
paper_authors.user: shallowdream204
paper_authors._id: 66ecd8f89cbe3098580d1a3a
paper_authors.name: Huaibo Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T07:52:50.524Z
paper_authors._id: 642259feeaad1bcb28b44f55
paper_authors.avatarUrl: /avatars/685791fc7e66d685e9446228c9e278b1.svg
paper_authors.fullname: Huaibo Huang
paper_authors.type: user
paper_authors.user: Bob072
paper_authors._id: 66ecd8f89cbe3098580d1a3b
paper_authors.name: Ran He
paper_authors._id: 66ecd8f89cbe3098580d1a3c
paper_authors.name: Zhenheng Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T07:53:14.014Z
paper_authors._id: 6421183b69a2c2933882d652
paper_authors.avatarUrl: /avatars/66813a8fa22915087cccd4dbfb945ca7.svg
paper_authors.fullname: Zhenheng Yang
paper_authors.type: user
paper_authors.user: zhenheny
paper_authors._id: 66ecd8f89cbe3098580d1a3d
paper_authors.name: Quanzeng You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T07:53:21.169Z
paper_authors._id: 642bb8c87f152f6e72b8d3cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642bb8c87f152f6e72b8d3cd/MXxTssocydbabGZ6XFIic.png
paper_authors.fullname: Quanzeng You
paper_authors.type: user
paper_authors.user: Ye27
paper_title: InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced
    Mathematical Reasoning
paper_summary: Pre-training on large-scale, high-quality datasets is crucial for enhancing
  the reasoning capabilities of Large Language Models (LLMs), especially in
  specialized domains such as mathematics. Despite the recognized importance, the
  Multimodal LLMs (MLLMs) field currently lacks a comprehensive open-source
  pre-training dataset specifically designed for mathematical reasoning. To
  address this gap, we introduce InfiMM-WebMath-40B, a high-quality dataset of
  interleaved image-text documents. It comprises 24 million web pages, 85 million
  associated image URLs, and 40 billion text tokens, all meticulously extracted
  and filtered from CommonCrawl. We provide a detailed overview of our data
  collection and processing pipeline. To demonstrate the robustness of
  InfiMM-WebMath-40B, we conducted evaluations in both text-only and multimodal
  settings. Our evaluations on text-only benchmarks show that, despite utilizing
  only 40 billion tokens, our dataset significantly enhances the performance of
  our 1.3B model, delivering results comparable to DeepSeekMath-1.3B, which uses
  120 billion tokens for the same model size. Nevertheless, with the introduction
  of our multi-modal math pre-training dataset, our models set a new
  state-of-the-art among open-source models on multi-modal math benchmarks such
  as MathVerse and We-Math. We release our data at
  https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.
paper_ai_keywords: Large Language Models (LLMs) Multimodal LLMs (MLLMs) Mathematical reasoning CommonCrawl Phrase embeddings Text-only benchmarks Multi-modal math pre-training dataset MathVerse We-Math DeepSeekMath-1.3B