Unicorn: Text-Only Data Synthesis for Vision Language Model Training
Training vision-language models (VLMs) typically requires large-scale,
high-quality image-text pairs, but collecting or synthesizing such data is
costly. In contrast, text data is abundant and inexpensive, prompting the
question: can high-quality multimodal training data be synthesized purely from
text? To tackle this, we propose a cross-integrated three-stage multimodal data
synthesis framework, which generates two datasets: Unicorn-1.2M and
Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we
construct 1.2M semantically diverse high-quality captions by expanding sparse
caption seeds using large language models (LLMs). In Stage 2:
Instruction-Tuning Data Generation, we further process 471K captions into
multi-turn instruction-tuning tasks to support complex reasoning. Finally, in
Stage 3: Modality Representation Transfer, these textual captions
representations are transformed into visual representations, resulting in
diverse synthetic image representations. This three-stage process enables us to
construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for
instruction-tuning, without relying on real images. By eliminating the
dependency on real images while maintaining data quality and diversity, our
framework offers a cost-effective and scalable solution for VLMs training. Code
is available at https://github.com/Yu-xm/Unicorn.git.
67ebe24d386032554c9e84fc
Xiaomin Yu
claimed_verified
2025-04-02T09:49:42.683Z
64084fa192033c150738e4f2
/avatars/dfff2216eb235c635e5abe6fda3084f0.svg
Yu_xm
user
Yu2020
67ebe24d386032554c9e84fd
Pengxiang Ding
67ebe24d386032554c9e84fe
Wenjie Zhang
67ebe24d386032554c9e84ff
Siteng Huang
67ebe24d386032554c9e8500
Songyang Gao
67ebe24d386032554c9e8501
Chengwei Qin
67ebe24d386032554c9e8502
Kejian Wu
67ebe24d386032554c9e8503
Zhaoxin Fan
67ebe24d386032554c9e8504
Ziyue Qiao
67ebe24d386032554c9e8505
Donglin Wang
Unicorn: Text-Only Data Synthesis for Vision Language Model Training
Training vision-language models (VLMs) typically requires large-scale,
high-quality image-text pairs, but collecting or synthesizing such data is
costly. In contrast, text data is abundant and inexpensive, prompting the
question: can high-quality multimodal training data be synthesized purely from
text? To tackle this, we propose a cross-integrated three-stage multimodal data
synthesis framework, which generates two datasets: Unicorn-1.2M and
Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we
construct 1.2M semantically diverse high-quality captions by expanding sparse
caption seeds using large language models (LLMs). In Stage 2:
Instruction-Tuning Data Generation, we further process 471K captions into
multi-turn instruction-tuning tasks to support complex reasoning. Finally, in
Stage 3: Modality Representation Transfer, these textual captions
representations are transformed into visual representations, resulting in
diverse synthetic image representations. This three-stage process enables us to
construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for
instruction-tuning, without relying on real images. By eliminating the
dependency on real images while maintaining data quality and diversity, our
framework offers a cost-effective and scalable solution for VLMs training. Code
is available at https://github.com/Yu-xm/Unicorn.git.
A three-stage framework synthesizes high-quality multimodal datasets using text to generate captions and transform them into visual representations, enabling cost-effective training of vision-language models.
vision-language models multimodal data synthesis large language models diverse caption data synthesis instruction-tuning data generation modality representation transfer Unicorn-1.2M Unicorn-471K-Instruction