title: MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal
    Dataset with One Trillion Tokens
summary: Multimodal interleaved datasets featuring free-form interleaved sequences of
  images and text are crucial for training frontier large multimodal models
  (LMMs). Despite the rapid progression of open-source LMMs, there remains a
  pronounced scarcity of large-scale, diverse open-source multimodal interleaved
  datasets. In response, we introduce MINT-1T, the most extensive and diverse
  open-source Multimodal INTerleaved dataset to date. MINT-1T comprises one
  trillion text tokens and three billion images, a 10x scale-up from existing
  open-source datasets. Additionally, we include previously untapped sources such
  as PDFs and ArXiv papers. As scaling multimodal interleaved datasets requires
  substantial engineering effort, sharing the data curation process and releasing
  the dataset greatly benefits the community. Our experiments show that LMMs
  trained on MINT-1T rival the performance of models trained on the previous
  leading dataset, OBELICS. Our data and code will be released at
  https://github.com/mlfoundations/MINT-1T.
paper_authors._id: 667143979ab144eb3aad5beb
paper_authors.name: Anas Awadalla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T21:26:54.637Z
paper_authors._id: 60f1ab3afe0d78e01037eeb1
paper_authors.avatarUrl: /avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
paper_authors.fullname: Anas Awadalla
paper_authors.type: user
paper_authors.user: anas-awadalla
paper_authors._id: 667143979ab144eb3aad5bec
paper_authors.name: Le Xue
paper_authors._id: 667143979ab144eb3aad5bed
paper_authors.name: Oscar Lo
paper_authors._id: 667143979ab144eb3aad5bee
paper_authors.name: Manli Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:21:11.917Z
paper_authors._id: 63bdcb129a15a3e94198e7d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
paper_authors.fullname: Manli Shu
paper_authors.type: user
paper_authors.user: Manli
paper_authors._id: 667143979ab144eb3aad5bef
paper_authors.name: Hannah Lee
paper_authors._id: 667143979ab144eb3aad5bf0
paper_authors.name: Etash Kumar Guha
paper_authors._id: 667143979ab144eb3aad5bf1
paper_authors.name: Matt Jordan
paper_authors._id: 667143979ab144eb3aad5bf2
paper_authors.name: Sheng Shen
paper_authors._id: 667143979ab144eb3aad5bf3
paper_authors.name: Mohamed Awadalla
paper_authors._id: 667143979ab144eb3aad5bf4
paper_authors.name: Silvio Savarese
paper_authors._id: 667143979ab144eb3aad5bf5
paper_authors.name: Caiming Xiong
paper_authors._id: 667143979ab144eb3aad5bf6
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:14:07.356Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_authors._id: 667143979ab144eb3aad5bf7
paper_authors.name: Yejin Choi
paper_authors._id: 667143979ab144eb3aad5bf8
paper_authors.name: Ludwig Schmidt
paper_title: MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal
    Dataset with One Trillion Tokens
paper_summary: Multimodal interleaved datasets featuring free-form interleaved sequences of
  images and text are crucial for training frontier large multimodal models
  (LMMs). Despite the rapid progression of open-source LMMs, there remains a
  pronounced scarcity of large-scale, diverse open-source multimodal interleaved
  datasets. In response, we introduce MINT-1T, the most extensive and diverse
  open-source Multimodal INTerleaved dataset to date. MINT-1T comprises one
  trillion text tokens and three billion images, a 10x scale-up from existing
  open-source datasets. Additionally, we include previously untapped sources such
  as PDFs and ArXiv papers. As scaling multimodal interleaved datasets requires
  substantial engineering effort, sharing the data curation process and releasing
  the dataset greatly benefits the community. Our experiments show that LMMs
  trained on MINT-1T rival the performance of models trained on the previous
  leading dataset, OBELICS. Our data and code will be released at
  https://github.com/mlfoundations/MINT-1T.
paper_ai_summary: MINT-1T, the largest open-source multimodal interleaved dataset featuring a trillion text tokens and three billion images, enhances the performance of large multimodal models.
paper_ai_keywords: multimodal interleaved datasets large multimodal models LMMs MINT-1T PDFs ArXiv papers data curation