title: RealSyn: An Effective and Scalable Multimodal Interleaved Document
    Transformation Paradigm
summary: After pre-training on extensive image-text pairs, Contrastive Language-Image
  Pre-training (CLIP) demonstrates promising performance on a wide variety of
  benchmarks. However, a substantial volume of non-paired data, such as
  multimodal interleaved documents, remains underutilized for vision-language
  representation learning. To fully leverage these unpaired documents, we
  initially establish a Real-World Data Extraction pipeline to extract
  high-quality images and texts. Then we design a hierarchical retrieval method
  to efficiently associate each image with multiple semantically relevant
  realistic texts. To further enhance fine-grained visual information, we propose
  an image semantic augmented generation module for synthetic text production.
  Furthermore, we employ a semantic balance sampling strategy to improve dataset
  diversity, enabling better learning of long-tail concepts. Based on these
  innovations, we construct RealSyn, a dataset combining realistic and synthetic
  texts, available in three scales: 15M, 30M, and 100M. Extensive experiments
  demonstrate that RealSyn effectively advances vision-language representation
  learning and exhibits strong scalability. Models pre-trained on RealSyn achieve
  state-of-the-art performance on multiple downstream tasks. To facilitate future
  research, the RealSyn dataset and pre-trained model weights are released at
  https://github.com/deepglint/RealSyn.
paper_authors._id: 67b545fd88527668fa8bcc14
paper_authors.name: Tiancheng Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:19:15.243Z
paper_authors._id: 6508712e7ee07e274b0f4c94
paper_authors.avatarUrl: /avatars/23fe5593b0bce36c2167c3142e57e0e9.svg
paper_authors.fullname: Tiancheng Gu
paper_authors.type: user
paper_authors.user: GaryGuuu
paper_authors._id: 67b545fd88527668fa8bcc15
paper_authors.name: Kaicheng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T14:37:43.180Z
paper_authors._id: 63e202f352b7578dba448ab5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
paper_authors.fullname: Yang
paper_authors.type: user
paper_authors.user: Kaichengalex
paper_authors._id: 67b545fd88527668fa8bcc16
paper_authors.name: Chaoyi Zhang
paper_authors._id: 67b545fd88527668fa8bcc17
paper_authors.name: Yin Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:04:06.471Z
paper_authors._id: 655c70d331c4978366d4b2e6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655c70d331c4978366d4b2e6/X-KjTNkxtzeYu9ngBOh_C.jpeg
paper_authors.fullname: Yiye
paper_authors.type: user
paper_authors.user: Yin-Xie
paper_authors._id: 67b545fd88527668fa8bcc18
paper_authors.name: Xiang An
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:39:00.405Z
paper_authors._id: 6478679d7b370854241b2ad8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6478679d7b370854241b2ad8/dBczWYYdfEt9tQcnVGhQk.jpeg
paper_authors.fullname: xiangan
paper_authors.type: user
paper_authors.user: xiangan
paper_authors._id: 67b545fd88527668fa8bcc19
paper_authors.name: Ziyong Feng
paper_authors._id: 67b545fd88527668fa8bcc1a
paper_authors.name: Dongnan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:20:00.745Z
paper_authors._id: 65e7f19e14856e8859fd8adc
paper_authors.avatarUrl: /avatars/203e919c361db94028a1b3c6ea52f0c2.svg
paper_authors.fullname: Dongnan Liu
paper_authors.type: user
paper_authors.user: Nina0607
paper_authors._id: 67b545fd88527668fa8bcc1b
paper_authors.name: Weidong Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:20:18.377Z
paper_authors._id: 6760a8f5e4b55ba1b2b0a7b4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NddUMmwmZFbS25v1q8KyS.png
paper_authors.fullname: Weidong Cai
paper_authors.type: user
paper_authors.user: SeriousBro
paper_authors._id: 67b545fd88527668fa8bcc1c
paper_authors.name: Jiankang Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:20:25.455Z
paper_authors._id: 62cc7a38376917c0223dd24b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1657566065867-noauth.png
paper_authors.fullname: JiankangDeng
paper_authors.type: user
paper_authors.user: JiankangDeng
paper_title: RealSyn: An Effective and Scalable Multimodal Interleaved Document
    Transformation Paradigm
paper_summary: After pre-training on extensive image-text pairs, Contrastive Language-Image
  Pre-training (CLIP) demonstrates promising performance on a wide variety of
  benchmarks. However, a substantial volume of non-paired data, such as
  multimodal interleaved documents, remains underutilized for vision-language
  representation learning. To fully leverage these unpaired documents, we
  initially establish a Real-World Data Extraction pipeline to extract
  high-quality images and texts. Then we design a hierarchical retrieval method
  to efficiently associate each image with multiple semantically relevant
  realistic texts. To further enhance fine-grained visual information, we propose
  an image semantic augmented generation module for synthetic text production.
  Furthermore, we employ a semantic balance sampling strategy to improve dataset
  diversity, enabling better learning of long-tail concepts. Based on these
  innovations, we construct RealSyn, a dataset combining realistic and synthetic
  texts, available in three scales: 15M, 30M, and 100M. Extensive experiments
  demonstrate that RealSyn effectively advances vision-language representation
  learning and exhibits strong scalability. Models pre-trained on RealSyn achieve
  state-of-the-art performance on multiple downstream tasks. To facilitate future
  research, the RealSyn dataset and pre-trained model weights are released at
  https://github.com/deepglint/RealSyn.
paper_ai_keywords: Contrastive Language-Image Pre-training (CLIP) pre-training image-text pairs non-paired data curated multimodal interleaved documents vision-language representation learning Real-World Data Extraction pipeline scheduling retrieval method image semantic augmented generation module synthetic text production semantic balance sampling strategy dataset diversity long-tail concepts multimodal representations RealSyn state-of-the-art performance downstream tasks pre-trained model weights