title: OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images
    Interleaved with Text
summary: Image-text interleaved data, consisting of multiple images and texts arranged
  in a natural document format, aligns with the presentation paradigm of internet
  data and closely resembles human reading habits. Recent studies have shown that
  such data aids multimodal in-context learning and maintains the capabilities of
  large language models during multimodal fine-tuning. However, the limited scale
  and diversity of current image-text interleaved data restrict the development
  of multimodal large language models. In this paper, we introduce OmniCorpus, a
  10 billion-scale image-text interleaved dataset. Using an efficient data
  engine, we filter and extract large-scale high-quality documents, which contain
  8.6 billion images and 1,696 billion text tokens. Compared to counterparts
  (e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while
  maintaining good data quality; 2) features more diverse sources, including both
  English and non-English websites as well as video-centric websites; 3) is more
  flexible, easily degradable from an image-text interleaved format to pure text
  corpus and image-text pairs. Through comprehensive analysis and experiments, we
  validate the quality, usability, and effectiveness of the proposed dataset. We
  hope this could provide a solid data foundation for future multimodal model
  research. Code and data are released at
  https://github.com/OpenGVLab/OmniCorpus.
paper_authors._id: 666fd32c4aae2409910e249b
paper_authors.name: Qingyun Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:54:12.447Z
paper_authors._id: 637f100c1dbae09191088e4c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669271529782-noauth.jpeg
paper_authors.fullname: Qingyun Li
paper_authors.type: user
paper_authors.user: Qingyun
paper_authors._id: 666fd32c4aae2409910e249c
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:49:27.780Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 666fd32c4aae2409910e249d
paper_authors.name: Weiyun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T21:27:39.234Z
paper_authors._id: 619507e7b74b6c591f794340
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
paper_authors.fullname: Weiyun Wang
paper_authors.type: user
paper_authors.user: Weiyun1025
paper_authors._id: 666fd32c4aae2409910e249e
paper_authors.name: Wenhai Wang
paper_authors._id: 666fd32c4aae2409910e249f
paper_authors.name: Shenglong Ye
paper_authors._id: 666fd32c4aae2409910e24a0
paper_authors.name: Zhenjiang Jin
paper_authors._id: 666fd32c4aae2409910e24a1
paper_authors.name: Guanzhou Chen
paper_authors._id: 666fd32c4aae2409910e24a2
paper_authors.name: Yinan He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:50:48.391Z
paper_authors._id: 62aafa49f29ff279b51f0182
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
paper_authors.fullname: yinanhe
paper_authors.type: user
paper_authors.user: ynhe
paper_authors._id: 666fd32c4aae2409910e24a3
paper_authors.name: Zhangwei Gao
paper_authors._id: 666fd32c4aae2409910e24a4
paper_authors.name: Erfei Cui
paper_authors._id: 666fd32c4aae2409910e24a5
paper_authors.name: Jiashuo Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-29T07:06:22.687Z
paper_authors._id: 64b89a14cf14c2fabe96664c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
paper_authors.fullname: Jiashuo Yu
paper_authors.type: user
paper_authors.user: awojustin
paper_authors._id: 666fd32c4aae2409910e24a6
paper_authors.name: Hao Tian
paper_authors._id: 666fd32c4aae2409910e24a7
paper_authors.name: Jiasheng Zhou
paper_authors._id: 666fd32c4aae2409910e24a8
paper_authors.name: Chao Xu
paper_authors._id: 666fd32c4aae2409910e24a9
paper_authors.name: Bin Wang
paper_authors._id: 666fd32c4aae2409910e24aa
paper_authors.name: Xingjian Wei
paper_authors._id: 666fd32c4aae2409910e24ab
paper_authors.name: Wei Li
paper_authors._id: 666fd32c4aae2409910e24ac
paper_authors.name: Wenjian Zhang
paper_authors._id: 666fd32c4aae2409910e24ad
paper_authors.name: Bo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:54:20.763Z
paper_authors._id: 643dfd235aafbdca3a5792c0
paper_authors.avatarUrl: /avatars/ce8553cf5936012c692e08054ee27937.svg
paper_authors.fullname: Bo Zhang
paper_authors.type: user
paper_authors.user: BoZhang
paper_authors._id: 666fd32c4aae2409910e24ae
paper_authors.name: Pinlong Cai
paper_authors._id: 666fd32c4aae2409910e24af
paper_authors.name: Licheng Wen
paper_authors._id: 666fd32c4aae2409910e24b0
paper_authors.name: Xiangchao Yan
paper_authors._id: 666fd32c4aae2409910e24b1
paper_authors.name: Zhenxiang Li
paper_authors._id: 666fd32c4aae2409910e24b2
paper_authors.name: Pei Chu
paper_authors._id: 666fd32c4aae2409910e24b3
paper_authors.name: Yi Wang
paper_authors._id: 666fd32c4aae2409910e24b4
paper_authors.name: Min Dou
paper_authors._id: 666fd32c4aae2409910e24b5
paper_authors.name: Changyao Tian
paper_authors._id: 666fd32c4aae2409910e24b6
paper_authors.name: Xizhou Zhu
paper_authors._id: 666fd32c4aae2409910e24b7
paper_authors.name: Lewei Lu
paper_authors._id: 666fd32c4aae2409910e24b8
paper_authors.name: Yushi Chen
paper_authors._id: 666fd32c4aae2409910e24b9
paper_authors.name: Junjun He
paper_authors._id: 666fd32c4aae2409910e24ba
paper_authors.name: Zhongying Tu
paper_authors._id: 666fd32c4aae2409910e24bb
paper_authors.name: Tong Lu
paper_authors._id: 666fd32c4aae2409910e24bc
paper_authors.name: Yali Wang
paper_authors._id: 666fd32c4aae2409910e24bd
paper_authors.name: Limin Wang
paper_authors._id: 666fd32c4aae2409910e24be
paper_authors.name: Dahua Lin
paper_authors._id: 666fd32c4aae2409910e24bf
paper_authors.name: Yu Qiao
paper_authors._id: 666fd32c4aae2409910e24c0
paper_authors.name: Botian Shi
paper_authors._id: 666fd32c4aae2409910e24c1
paper_authors.name: Conghui He
paper_authors._id: 666fd32c4aae2409910e24c2
paper_authors.name: Jifeng Dai
paper_title: OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images
    Interleaved with Text
paper_summary: Image-text interleaved data, consisting of multiple images and texts arranged
  in a natural document format, aligns with the presentation paradigm of internet
  data and closely resembles human reading habits. Recent studies have shown that
  such data aids multimodal in-context learning and maintains the capabilities of
  large language models during multimodal fine-tuning. However, the limited scale
  and diversity of current image-text interleaved data restrict the development
  of multimodal large language models. In this paper, we introduce OmniCorpus, a
  10 billion-scale image-text interleaved dataset. Using an efficient data
  engine, we filter and extract large-scale high-quality documents, which contain
  8.6 billion images and 1,696 billion text tokens. Compared to counterparts
  (e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while
  maintaining good data quality; 2) features more diverse sources, including both
  English and non-English websites as well as video-centric websites; 3) is more
  flexible, easily degradable from an image-text interleaved format to pure text
  corpus and image-text pairs. Through comprehensive analysis and experiments, we
  validate the quality, usability, and effectiveness of the proposed dataset. We
  hope this could provide a solid data foundation for future multimodal model
  research. Code and data are released at
  https://github.com/OpenGVLab/OmniCorpus.
paper_ai_summary: OmniCorpus, a large-scale image-text interleaved dataset, enhances multimodal model research by increasing volume, diversity, and flexibility over existing datasets.
paper_ai_keywords: multimodal in-context learning multimodal fine-tuning large language models image-text interleaved dataset data engine data quality image-text format image-text pairs