title: Unicorn: Text-Only Data Synthesis for Vision Language Model Training
summary: Training vision-language models (VLMs) typically requires large-scale,
  high-quality image-text pairs, but collecting or synthesizing such data is
  costly. In contrast, text data is abundant and inexpensive, prompting the
  question: can high-quality multimodal training data be synthesized purely from
  text? To tackle this, we propose a cross-integrated three-stage multimodal data
  synthesis framework, which generates two datasets: Unicorn-1.2M and
  Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we
  construct 1.2M semantically diverse high-quality captions by expanding sparse
  caption seeds using large language models (LLMs). In Stage 2:
  Instruction-Tuning Data Generation, we further process 471K captions into
  multi-turn instruction-tuning tasks to support complex reasoning. Finally, in
  Stage 3: Modality Representation Transfer, these textual captions
  representations are transformed into visual representations, resulting in
  diverse synthetic image representations. This three-stage process enables us to
  construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for
  instruction-tuning, without relying on real images. By eliminating the
  dependency on real images while maintaining data quality and diversity, our
  framework offers a cost-effective and scalable solution for VLMs training. Code
  is available at https://github.com/Yu-xm/Unicorn.git.
paper_authors._id: 67ebe24d386032554c9e84fc
paper_authors.name: Xiaomin Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T09:49:42.683Z
paper_authors._id: 64084fa192033c150738e4f2
paper_authors.avatarUrl: /avatars/dfff2216eb235c635e5abe6fda3084f0.svg
paper_authors.fullname: Yu_xm
paper_authors.type: user
paper_authors.user: Yu2020
paper_authors._id: 67ebe24d386032554c9e84fd
paper_authors.name: Pengxiang Ding
paper_authors._id: 67ebe24d386032554c9e84fe
paper_authors.name: Wenjie Zhang
paper_authors._id: 67ebe24d386032554c9e84ff
paper_authors.name: Siteng Huang
paper_authors._id: 67ebe24d386032554c9e8500
paper_authors.name: Songyang Gao
paper_authors._id: 67ebe24d386032554c9e8501
paper_authors.name: Chengwei Qin
paper_authors._id: 67ebe24d386032554c9e8502
paper_authors.name: Kejian Wu
paper_authors._id: 67ebe24d386032554c9e8503
paper_authors.name: Zhaoxin Fan
paper_authors._id: 67ebe24d386032554c9e8504
paper_authors.name: Ziyue Qiao
paper_authors._id: 67ebe24d386032554c9e8505
paper_authors.name: Donglin Wang
paper_title: Unicorn: Text-Only Data Synthesis for Vision Language Model Training
paper_summary: Training vision-language models (VLMs) typically requires large-scale,
  high-quality image-text pairs, but collecting or synthesizing such data is
  costly. In contrast, text data is abundant and inexpensive, prompting the
  question: can high-quality multimodal training data be synthesized purely from
  text? To tackle this, we propose a cross-integrated three-stage multimodal data
  synthesis framework, which generates two datasets: Unicorn-1.2M and
  Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we
  construct 1.2M semantically diverse high-quality captions by expanding sparse
  caption seeds using large language models (LLMs). In Stage 2:
  Instruction-Tuning Data Generation, we further process 471K captions into
  multi-turn instruction-tuning tasks to support complex reasoning. Finally, in
  Stage 3: Modality Representation Transfer, these textual captions
  representations are transformed into visual representations, resulting in
  diverse synthetic image representations. This three-stage process enables us to
  construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for
  instruction-tuning, without relying on real images. By eliminating the
  dependency on real images while maintaining data quality and diversity, our
  framework offers a cost-effective and scalable solution for VLMs training. Code
  is available at https://github.com/Yu-xm/Unicorn.git.
paper_ai_summary: A three-stage framework synthesizes high-quality multimodal datasets using text to generate captions and transform them into visual representations, enabling cost-effective training of vision-language models.
paper_ai_keywords: vision-language models multimodal data synthesis large language models diverse caption data synthesis instruction-tuning data generation modality representation transfer Unicorn-1.2M Unicorn-471K-Instruction