RealSyn: An Effective and Scalable Multimodal Interleaved Document
  Transformation Paradigm
After pre-training on extensive image-text pairs, Contrastive Language-Image
Pre-training (CLIP) demonstrates promising performance on a wide variety of
benchmarks. However, a substantial volume of non-paired data, such as
multimodal interleaved documents, remains underutilized for vision-language
representation learning. To fully leverage these unpaired documents, we
initially establish a Real-World Data Extraction pipeline to extract
high-quality images and texts. Then we design a hierarchical retrieval method
to efficiently associate each image with multiple semantically relevant
realistic texts. To further enhance fine-grained visual information, we propose
an image semantic augmented generation module for synthetic text production.
Furthermore, we employ a semantic balance sampling strategy to improve dataset
diversity, enabling better learning of long-tail concepts. Based on these
innovations, we construct RealSyn, a dataset combining realistic and synthetic
texts, available in three scales: 15M, 30M, and 100M. Extensive experiments
demonstrate that RealSyn effectively advances vision-language representation
learning and exhibits strong scalability. Models pre-trained on RealSyn achieve
state-of-the-art performance on multiple downstream tasks. To facilitate future
research, the RealSyn dataset and pre-trained model weights are released at
https://github.com/deepglint/RealSyn.
67b545fd88527668fa8bcc14
Tiancheng Gu
claimed_verified
2025-04-25T08:36:16.350Z
641030c77a15af878ae5bd8f
/avatars/8a5037edf55c78ebc317c8b191343671.svg
TianchengGu
user
TianchengGu
67b545fd88527668fa8bcc15
Kaicheng Yang
claimed_verified
2025-02-19T14:37:43.180Z
63e202f352b7578dba448ab5
https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
Yang
user
Kaichengalex
67b545fd88527668fa8bcc16
Chaoyi Zhang
67b545fd88527668fa8bcc17
Yin Xie
claimed_verified
2025-03-10T08:04:06.471Z
655c70d331c4978366d4b2e6
https://cdn-avatars.huggingface.co/v1/production/uploads/655c70d331c4978366d4b2e6/X-KjTNkxtzeYu9ngBOh_C.jpeg
Yiye
user
Yin-Xie
67b545fd88527668fa8bcc18
Xiang An
claimed_verified
2025-02-26T08:39:00.405Z
6478679d7b370854241b2ad8
https://cdn-avatars.huggingface.co/v1/production/uploads/6478679d7b370854241b2ad8/dBczWYYdfEt9tQcnVGhQk.jpeg
xiangan
user
xiangan
67b545fd88527668fa8bcc19
Ziyong Feng
67b545fd88527668fa8bcc1a
Dongnan Liu
admin_assigned
2025-02-19T10:20:00.745Z
65e7f19e14856e8859fd8adc
/avatars/203e919c361db94028a1b3c6ea52f0c2.svg
Dongnan Liu
user
Nina0607
67b545fd88527668fa8bcc1b
Weidong Cai
admin_assigned
2025-02-19T10:20:18.377Z
6760a8f5e4b55ba1b2b0a7b4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NddUMmwmZFbS25v1q8KyS.png
Weidong Cai
user
SeriousBro
67b545fd88527668fa8bcc1c
Jiankang Deng
admin_assigned
2025-02-19T10:20:25.455Z
62cc7a38376917c0223dd24b
https://cdn-avatars.huggingface.co/v1/production/uploads/1657566065867-noauth.png
JiankangDeng
user
JiankangDeng
RealSyn: An Effective and Scalable Multimodal Interleaved Document
  Transformation Paradigm
After pre-training on extensive image-text pairs, Contrastive Language-Image
Pre-training (CLIP) demonstrates promising performance on a wide variety of
benchmarks. However, a substantial volume of non-paired data, such as
multimodal interleaved documents, remains underutilized for vision-language
representation learning. To fully leverage these unpaired documents, we
initially establish a Real-World Data Extraction pipeline to extract
high-quality images and texts. Then we design a hierarchical retrieval method
to efficiently associate each image with multiple semantically relevant
realistic texts. To further enhance fine-grained visual information, we propose
an image semantic augmented generation module for synthetic text production.
Furthermore, we employ a semantic balance sampling strategy to improve dataset
diversity, enabling better learning of long-tail concepts. Based on these
innovations, we construct RealSyn, a dataset combining realistic and synthetic
texts, available in three scales: 15M, 30M, and 100M. Extensive experiments
demonstrate that RealSyn effectively advances vision-language representation
learning and exhibits strong scalability. Models pre-trained on RealSyn achieve
state-of-the-art performance on multiple downstream tasks. To facilitate future
research, the RealSyn dataset and pre-trained model weights are released at
https://github.com/deepglint/RealSyn.
RealSyn dataset, combining realistic and synthetic texts, enhances vision-language representation learning by leveraging unpaired multimodal data.
Contrastive Language-Image Pre-training (CLIP) multimodal interleaved documents Real-World Data Extraction hierarchical retrieval method image semantic augmented generation module semantic balance sampling strategy vision-language representation learning state-of-the-art performance